Sep 16 18:43:30.507556 ip-10-0-142-173 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:43:30.507566 ip-10-0-142-173 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:43:30.507574 ip-10-0-142-173 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:43:30.507809 ip-10-0-142-173 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:43:40.756554 ip-10-0-142-173 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:43:40.756570 ip-10-0-142-173 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 992dd5fcc8004a39b91827b668058620 --
Sep 16 18:45:42.047930 ip-10-0-142-173 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:45:42.454191 ip-10-0-142-173 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:42.454191 ip-10-0-142-173 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:45:42.454191 ip-10-0-142-173 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:42.454191 ip-10-0-142-173 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:45:42.454191 ip-10-0-142-173 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:42.455020 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.454264    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:45:42.460798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460761    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:42.460798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460789    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:42.460798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460796    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:42.460798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460800    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:42.460798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460805    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:42.460798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460809    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460814    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460818    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460822    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460827    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460833    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460837    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460848    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460852    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460857    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460861    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460865    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460869    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460873    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460878    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460883    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460887    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460891    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460895    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460898    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:42.461177 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460902    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460906    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460909    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460914    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460918    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460921    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460925    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460930    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460934    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460938    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460942    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460945    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460949    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460954    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460959    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460964    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460968    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460972    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460976    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460981    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:42.462075 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460985    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460989    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460993    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.460997    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461001    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461010    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461014    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461018    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461023    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461027    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461031    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461035    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461040    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461044    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461048    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461052    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461056    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461060    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461064    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461068    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:42.462956 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461072    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461076    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461080    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461084    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461088    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461094    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461098    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461103    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461107    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461112    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461116    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461120    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461124    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461128    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461134    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461138    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461142    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461146    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461155    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:42.463798 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461162    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461166    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461171    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461175    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461178    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461182    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461186    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461190    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461194    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461198    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461203    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461207    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461211    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461215    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.461227    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462460    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462472    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462476    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462481    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462486    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:42.464667 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462490    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462496    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462499    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462505    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462509    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462513    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462518    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462522    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462526    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462531    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462535    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462540    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462545    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462550    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462554    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462558    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462563    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462567    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462591    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462595    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:42.465530 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462599    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462604    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462609    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462613    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462618    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462622    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462628    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462632    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462637    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462641    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462646    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462652    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462656    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462660    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462664    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462669    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462673    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462678    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462682    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462686    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:42.466213 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462690    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462695    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462699    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462703    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462707    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462713    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462718    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462722    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462726    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462731    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462735    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462739    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462743    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462747    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462752    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462756    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462761    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462765    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462772    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462778    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:42.466866 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462782    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462788    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462793    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462797    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462802    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462807    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462811    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462816    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462820    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462824    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462828    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462833    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462840    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462845    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462849    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462854    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462859    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462863    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462873    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:42.467452 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462877    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462883    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462888    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462893    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462897    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462901    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462906    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462912    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462917    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462922    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462927    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462932    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462936    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462941    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.462945    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463641    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463652    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463656    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463661    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:42.468039 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463666    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463670    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463675    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463679    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463683    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463688    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463692    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463697    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463701    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463706    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463710    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463715    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463725    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463730    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463734    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463738    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463742    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463746    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463751    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463755    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:42.468746 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463759    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463764    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463768    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463775    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463781    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463786    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463791    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463796    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463801    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463805    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463809    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463814    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463818    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463824    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463829    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463834    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463839    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463843    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463847    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463852    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:42.469310 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463857    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463864    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463883    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463888    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463892    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463898    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463903    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463907    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463911    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463916    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463920    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463924    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463928    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463932    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463936    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463940    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463944    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463949    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463952    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:42.469854 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463958    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463962    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463966    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463971    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463975    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463980    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463984    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463988    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463992    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.463996    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464000    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464004    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464008    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464011    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464016    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464020    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464024    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464029    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464033    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464038    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:42.470405 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464043    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464047    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464051    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464054    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464059    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464063    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464067    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464074    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464078    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464083    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464087    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464091    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464096    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464100    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464104    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.464108    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464230    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464288    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464298    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464305    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464313    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:45:42.471011 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464319    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464326    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464333    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464338    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464343    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464349    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464354    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464359    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464365    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464370    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464377    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464384    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464390    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464397    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464402    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464407    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464412    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464419    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464426    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464431    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464437    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464442    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464451    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464456    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:45:42.471519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464461    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464467    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464472    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464479    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464484    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464489    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464494    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464498    2582 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464503    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464509    2582 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464514    2582 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464520    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464524    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464530    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464536    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464541    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464547    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464552    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464557    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464562    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464567    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464593    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464598    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464603    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464608    2582 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:45:42.472217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464615    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464620    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464626    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464631    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464636    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464643    2582 flags.go:64] FLAG: --help="false"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464648    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464655    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464660    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464665    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464670    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464676    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464681    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464686    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464690    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464694    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464700    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464706    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464711    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464716    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464721    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464728    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464736    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464742    2582 flags.go:64] FLAG: --lock-file=""
Sep 16 18:45:42.472885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464747    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464751    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464756    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464766    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464772    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464778    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464783    2582 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464788    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464793    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464798    2582 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464802    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464810    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464815    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464826    2582 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464831    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464837    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464842    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464848    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464853    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464858    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464862    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464878    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464883    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464888    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:45:42.473526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464893    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464898    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464906    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464911    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464916    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464920    2582 flags.go:64] FLAG: --port="10250"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464928    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464933    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e190d1bd9c0cfbc9"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464938    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464942    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464947    2582 flags.go:64] FLAG: --register-node="true"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464952    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464959    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464966    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464971    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464976    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464981    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464987    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464992    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.464997    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465002    2582 flags.go:64] FLAG: --runonce="false"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465007    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465014    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465019    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465024    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:45:42.474186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465028    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465033    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465045    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465050    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465055    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465060    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465065    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465070    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465075    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465080    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465085    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465095    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465100    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465107    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465116    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465120    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465125    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465129    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465134    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465138    2582 flags.go:64] FLAG: --v="2"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465145    2582 flags.go:64] FLAG: --version="false"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465157    2582 flags.go:64] FLAG: --vmodule=""
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465164    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.465169    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465367    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:42.474893 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465375    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465379    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465384    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465389    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465396    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465402    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465408    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465413    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465418    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465422    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465426    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465431    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465435    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465440    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465444    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465448    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465451    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465456    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465461    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:42.475513 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465466    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465472    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465476    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465481    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465485    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465489    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465493    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465498    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465503    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465507    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465512    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465517    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465521    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465526    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465529    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465533    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465537    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465544    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465548    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465553    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:42.476022 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465557    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465562    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465567    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465589    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465593    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465598    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465602    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465606    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465610    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465615    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465619    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465626    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465631    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465639    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465645    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465650    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465655    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465659    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465663    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:42.476524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465668    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465672    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465677    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465681    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465687    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465691    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465695    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465700    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465704    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465708    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465715    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465719    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465724    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465728    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465732    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465736    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465740    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465744    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465749    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465753    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:42.477025 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465758    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465764    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465771    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465776    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465781    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465786    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465792    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465796    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465800    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465804    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465809    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465813    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465817    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465821    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465825    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465830    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465834    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465838    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465843    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:42.477524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.465847    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:42.478009 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.466480    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:42.478009 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.475173    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:45:42.478009 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.475191    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:45:42.478009 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.475208    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:45:42.478009 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.475214    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:45:42.478009 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475280    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:42.478009 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475287    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:42.478009 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475290    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:42.478009 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475294    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:42.478009 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475299    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:42.478009 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475302    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:42.478009 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475306    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:42.478009 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475310    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:42.478009 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475313    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475316    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475319    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475322    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475325    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475328    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475331    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475334    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475337    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475339    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475342    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475345    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475347    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475350    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475353    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475355    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475358    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475361    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475363    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475366    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:42.478367 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475368    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475374    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475376    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475379    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475381    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475384    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475387    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475389    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475392    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475394    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475397    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475399    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475402    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475405    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475407    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475410    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475413    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475415    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475418    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475421    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:42.478889 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475423    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475427    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475431    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475433    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475437    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475439    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475442    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475444    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475447    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475449    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475452    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475455    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475457    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475461    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475470    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475473    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475476    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475478    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475481    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:42.479384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475484    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475494    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475496    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475499    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475502    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475504    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475507    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475510    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475512    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475514    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475517    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475520    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475522    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475525    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475528    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475530    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475533    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475535    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475538    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:42.479959 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475541    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475544    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475546    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475549    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475552    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475554    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475557    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475559    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475563    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475584    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475588    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475590    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475593    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.475599    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475753    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:42.480433 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475760    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475763    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475766    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475769    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475772    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475775    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475777    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475780    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475783    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475786    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475790    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475793    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475795    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475798    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475801    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475803    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475807    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475812    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475815    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475818    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:42.480841 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475821    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475823    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475826    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475829    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475831    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475834    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475836    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475846    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475848    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475851    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475853    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475856    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475858    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475861    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475863    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475866    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475882    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475885    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475888    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475890    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:42.481339 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475893    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475896    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475899    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475902    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475905    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475907    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475911    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475915    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475917    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475920    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475924    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475927    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475929    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475932    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475935    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475938    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475940    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475944    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475946    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:42.481842 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475949    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475958    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475961    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475963    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475966    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475969    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475971    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475974    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475977    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475980    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475983    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475985    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475988    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475991    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475994    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475996    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.475999    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476001    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476004    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476006    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:42.482306 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476008    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476011    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476014    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476017    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476019    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476022    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476025    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476028    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476030    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476033    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476035    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476038    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476041    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476044    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476052    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476055    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476057    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476060    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:42.482821 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:42.476062    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:42.483256 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.476067    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:42.483256 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.476208    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:45:42.483256 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.479618    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:45:42.483256 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.480748    2582 server.go:1019] "Starting client certificate rotation"
Sep 16 18:45:42.483256 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.480873    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:42.483256 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.480940    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:42.483709 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.483607    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:42.492186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.492158    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:42.506126 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.506101    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:45:42.513368 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.513339    2582 log.go:25] "Validated CRI v1 image API"
Sep 16 18:45:42.517126 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.517104    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:42.518075 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.518058    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:45:42.521897 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.521874    2582 fs.go:135] Filesystem UUIDs: map[505493e1-aad9-43c5-9069-d36dc7c6f3b1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ab376fd5-ad04-4ba3-8432-b66cebf90298:/dev/nvme0n1p4]
Sep 16 18:45:42.521969 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.521896    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:45:42.527272 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.527144    2582 manager.go:216] Machine: {Timestamp:2026-09-16 18:45:42.525810544 +0000 UTC m=+0.356419080 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3086549 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec222cd976426f490f01bb522fd44438 SystemUUID:ec222cd9-7642-6f49-0f01-bb522fd44438 BootID:992dd5fc-c800-4a39-b918-27b668058620 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d9:06:4a:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d9:06:4a:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:6c:f6:09:88:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:45:42.527272 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.527264    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:45:42.527425 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.527354    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:45:42.530791 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.530761    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:45:42.530944 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.530794    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-173.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:45:42.531693 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.531681    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:45:42.531737 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.531696    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:45:42.531737 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.531705    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:42.533117 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.533103    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:42.534020 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.534010    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:45:42.537202 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.537189    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:42.537349 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.537340    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:45:42.537381 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.537374    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:45:42.537408 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.537386    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:45:42.537408 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.537402    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:45:42.537461 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.537411    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:45:42.537461 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.537423    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:45:42.537898 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.537883    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:42.537968 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.537916    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:42.540066 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.540044    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:45:42.541035 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.541017    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5ljzj"
Sep 16 18:45:42.542541 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.542521    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:45:42.542659 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.542560    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:45:42.544283 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.544269    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:45:42.544370 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.544289    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:45:42.544370 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.544299    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:45:42.544370 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.544308    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:45:42.544370 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.544320    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:45:42.544370 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.544330    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:45:42.544370 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.544349    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:45:42.544370 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.544358    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:45:42.544370 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.544369    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:45:42.544675 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.544378    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:45:42.544675 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.544392    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:45:42.544675 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.544401    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:45:42.545102 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.545089    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:45:42.545149 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.545106    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:45:42.546834 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.546812    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5ljzj"
Sep 16 18:45:42.547254 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.547233    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-173.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:45:42.547392 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.547368    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:45:42.547554 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.547536    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:45:42.549382 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.549367    2582 server.go:1268] "Started kubelet"
Sep 16 18:45:42.549491 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.549458    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:45:42.549730 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.549637    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:45:42.552077 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.551844    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:45:42.552069 ip-10-0-142-173 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:45:42.554105 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.554086    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:45:42.554462 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.554256    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:45:42.556801 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.556760    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:45:42.557903 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.556970    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-173.ec2.internal.18d5e17a594123e0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-173.ec2.internal,UID:ip-10-0-142-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-173.ec2.internal,},FirstTimestamp:2026-09-16 18:45:42.549341152 +0000 UTC m=+0.379949689,LastTimestamp:2026-09-16 18:45:42.549341152 +0000 UTC m=+0.379949689,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-173.ec2.internal,}"
Sep 16 18:45:42.559804 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.559788    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:42.560470 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.560452    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:45:42.561017 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.560716    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:42.561184 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.561163    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:45:42.561184 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.561165    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:45:42.561332 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.561209    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:45:42.561332 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.561292    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:45:42.561332 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.561304    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:45:42.561481 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.561448    2582 factory.go:153] Registering CRI-O factory
Sep 16 18:45:42.561481 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.561465    2582 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:45:42.561588 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.561512    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:45:42.561588 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.561522    2582 factory.go:55] Registering systemd factory
Sep 16 18:45:42.561588 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.561529    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:45:42.561588 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.561542    2582 factory.go:103] Registering Raw factory
Sep 16 18:45:42.561588 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.561562    2582 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:45:42.562002 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.561988    2582 manager.go:349] Starting recovery of all containers
Sep 16 18:45:42.571332 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.571308    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:42.572559 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.572541    2582 manager.go:354] Recovery completed
Sep 16 18:45:42.573425 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.573400    2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-173.ec2.internal\" not found" node="ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.577266 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.577249    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:42.581817 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.581799    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:42.581919 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.581832    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:42.581919 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.581852    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:42.582401 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.582381    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:45:42.582401 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.582400    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:45:42.582477 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.582417    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:42.582477 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.582437    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:42.583759 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.583743    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:42.583828 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.583778    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:42.583828 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.583794    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:42.583915 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.583843    2582 policy_none.go:49] "None policy: Start"
Sep 16 18:45:42.583915 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.583858    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:45:42.583915 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.583885    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:45:42.585246 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.585224    2582 policy_none.go:47] "Start"
Sep 16 18:45:42.636215 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.636197    2582 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:45:42.636342 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.636257    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:45:42.636342 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.636272    2582 server.go:87] "Starting device plugin registration server"
Sep 16 18:45:42.636710 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.636697    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:45:42.636792 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.636710    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:45:42.636856 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.636821    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:45:42.636928 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.636915    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:45:42.636982 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.636931    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:45:42.637516 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.637490    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:45:42.637635 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.637548    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:42.695715 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.695675    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:45:42.697204 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.697176    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:45:42.697357 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.697240    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:45:42.697357 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.697266    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:45:42.697357 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.697303    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:45:42.699031 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.699012    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:42.737630 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.737494    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:42.738566 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.738550    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:42.738657 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.738608    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:42.738657 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.738624    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:42.738657 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.738648    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.746627 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.746608    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.746707 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.746635    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-173.ec2.internal\": node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:42.764652 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.764628    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:42.797792 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.797747    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-173.ec2.internal"]
Sep 16 18:45:42.797953 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.797838    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:42.799570 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.799555    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:42.799647 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.799601    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:42.799647 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.799618    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:42.801115 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.801102    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:42.801254 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.801239    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.801297 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.801271    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:42.802649 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.802632    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:42.802720 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.802661    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:42.802720 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.802671    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:42.802720 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.802632    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:42.802823 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.802740    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:42.802823 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.802754    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:42.804142 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.804123    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.804231 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.804152    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:42.804877 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.804857    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:42.804964 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.804886    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:42.804964 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.804897    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:42.831607 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.831545    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-173.ec2.internal\" not found" node="ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.836362 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.836342    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-173.ec2.internal\" not found" node="ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.863300 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.863260    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f14cf8cd09ed7a1368917ccea4b43bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal\" (UID: \"2f14cf8cd09ed7a1368917ccea4b43bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.863300 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.863301    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f14cf8cd09ed7a1368917ccea4b43bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal\" (UID: \"2f14cf8cd09ed7a1368917ccea4b43bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.863521 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.863323    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a86b7e4f585f41568b1e18c5a2fe21b4-config\") pod \"kube-apiserver-proxy-ip-10-0-142-173.ec2.internal\" (UID: \"a86b7e4f585f41568b1e18c5a2fe21b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.865354 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.865334    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:42.963828 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.963771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f14cf8cd09ed7a1368917ccea4b43bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal\" (UID: \"2f14cf8cd09ed7a1368917ccea4b43bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.963828 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.963822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f14cf8cd09ed7a1368917ccea4b43bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal\" (UID: \"2f14cf8cd09ed7a1368917ccea4b43bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.964093 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.963874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a86b7e4f585f41568b1e18c5a2fe21b4-config\") pod \"kube-apiserver-proxy-ip-10-0-142-173.ec2.internal\" (UID: \"a86b7e4f585f41568b1e18c5a2fe21b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.964093 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.963916    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f14cf8cd09ed7a1368917ccea4b43bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal\" (UID: \"2f14cf8cd09ed7a1368917ccea4b43bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.964093 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.963929    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a86b7e4f585f41568b1e18c5a2fe21b4-config\") pod \"kube-apiserver-proxy-ip-10-0-142-173.ec2.internal\" (UID: \"a86b7e4f585f41568b1e18c5a2fe21b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.964093 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:42.963948    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f14cf8cd09ed7a1368917ccea4b43bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal\" (UID: \"2f14cf8cd09ed7a1368917ccea4b43bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:42.965851 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:42.965832    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:43.066845 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:43.066721    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:43.135987 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:43.135933    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:43.140843 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:43.140201    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:43.167550 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:43.167516    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:43.268074 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:43.268029    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:43.368640 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:43.368525    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:43.469090 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:43.469050    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:43.480238 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:43.480208    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:45:43.480397 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:43.480374    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:45:43.480435 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:43.480403    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:45:43.552522 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:43.552482    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:40:43 +0000 UTC" deadline="2028-05-03 15:34:53.347575977 +0000 UTC"
Sep 16 18:45:43.552522 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:43.552516    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14276h49m9.795063533s"
Sep 16 18:45:43.569769 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:43.569742    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:43.670761 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:43.670665    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:43.703743 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:43.703705    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f14cf8cd09ed7a1368917ccea4b43bd.slice/crio-89286d1686d52a48905b2ebc606ffb15fb078d7712a93febc74ac9fbb1b2a0cf WatchSource:0}: Error finding container 89286d1686d52a48905b2ebc606ffb15fb078d7712a93febc74ac9fbb1b2a0cf: Status 404 returned error can't find the container with id 89286d1686d52a48905b2ebc606ffb15fb078d7712a93febc74ac9fbb1b2a0cf
Sep 16 18:45:43.710377 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:43.710354    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:45:43.771776 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:43.771735    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:43.792211 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:43.792177    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86b7e4f585f41568b1e18c5a2fe21b4.slice/crio-f9713311e4dba1cab0acd91c049a864b69d78bdca6d7b32d388b3d33347996e9 WatchSource:0}: Error finding container f9713311e4dba1cab0acd91c049a864b69d78bdca6d7b32d388b3d33347996e9: Status 404 returned error can't find the container with id f9713311e4dba1cab0acd91c049a864b69d78bdca6d7b32d388b3d33347996e9
Sep 16 18:45:43.809867 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:43.809840    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:43.872849 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:43.872792    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-173.ec2.internal\" not found"
Sep 16 18:45:43.918106 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:43.918068    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:43.963628 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:43.963516    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:43.978428 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:43.978399    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:43.979674 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:43.979656    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-173.ec2.internal"
Sep 16 18:45:43.988923 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:43.988896    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:44.538931 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.538884    2582 apiserver.go:52] "Watching apiserver"
Sep 16 18:45:44.554594 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.554517    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:45:44.556404 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.556374    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-4twzx","openshift-multus/network-metrics-daemon-2wlpl","openshift-ovn-kubernetes/ovnkube-node-dvn2p","kube-system/global-pull-secret-syncer-bdq8w","kube-system/kube-apiserver-proxy-ip-10-0-142-173.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal","openshift-multus/multus-4tbdg","openshift-network-diagnostics/network-check-target-jsfns","openshift-network-operator/iptables-alerter-jqjj6","kube-system/konnectivity-agent-glsm5","openshift-cluster-node-tuning-operator/tuned-wtwdx","openshift-dns/node-resolver-stzn8","openshift-image-registry/node-ca-z9b6q"]
Sep 16 18:45:44.558787 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.558745    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jqjj6"
Sep 16 18:45:44.561172 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.561143    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:44.561941 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.561908    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:44.562093 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.561908    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:45:44.562168 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.562092    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-glsm5"
Sep 16 18:45:44.562383 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.562364    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:44.562519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.562485    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r7sbp\""
Sep 16 18:45:44.564354 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.564327    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.565158 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.565120    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xn7fs\""
Sep 16 18:45:44.565837 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.565815    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:45:44.565946 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.565860    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:45:44.566284 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.566262    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:44.566392 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:44.566362    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:45:44.567525 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.567506    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:45:44.568054 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.567668    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:45:44.568054 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.567763    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:45:44.568054 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.567927    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:45:44.568054 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.568013    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z9b6q"
Sep 16 18:45:44.568977 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.568856    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-28s8z\""
Sep 16 18:45:44.568977 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.568891    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:45:44.569086 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.569053    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:45:44.570102 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.570078    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.571291 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.571271    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:45:44.571432 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.571411    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p2l67\""
Sep 16 18:45:44.571756 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.571687    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:45:44.571756 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.571719    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:45:44.573122 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.573099    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:45:44.573239 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.573153    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:45:44.573479 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.573459    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tnt8m\""
Sep 16 18:45:44.573570 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.573468    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:45:44.574236 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.574205    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:45:44.575320 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:44.574291    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:45:44.575320 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.574549    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9-agent-certs\") pod \"konnectivity-agent-glsm5\" (UID: \"6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9\") " pod="kube-system/konnectivity-agent-glsm5"
Sep 16 18:45:44.575320 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.574585    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9-konnectivity-ca\") pod \"konnectivity-agent-glsm5\" (UID: \"6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9\") " pod="kube-system/konnectivity-agent-glsm5"
Sep 16 18:45:44.575320 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.574777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd6df1fa-d3f7-4f42-8476-4a1d8480e14d-iptables-alerter-script\") pod \"iptables-alerter-jqjj6\" (UID: \"fd6df1fa-d3f7-4f42-8476-4a1d8480e14d\") " pod="openshift-network-operator/iptables-alerter-jqjj6"
Sep 16 18:45:44.575320 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.574801    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:44.575320 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.574826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd6df1fa-d3f7-4f42-8476-4a1d8480e14d-host-slash\") pod \"iptables-alerter-jqjj6\" (UID: \"fd6df1fa-d3f7-4f42-8476-4a1d8480e14d\") " pod="openshift-network-operator/iptables-alerter-jqjj6"
Sep 16 18:45:44.575320 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.574875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfxb5\" (UniqueName: \"kubernetes.io/projected/fd6df1fa-d3f7-4f42-8476-4a1d8480e14d-kube-api-access-vfxb5\") pod \"iptables-alerter-jqjj6\" (UID: \"fd6df1fa-d3f7-4f42-8476-4a1d8480e14d\") " pod="openshift-network-operator/iptables-alerter-jqjj6"
Sep 16 18:45:44.576446 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.576410    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-stzn8"
Sep 16 18:45:44.578696 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.578674    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:44.578797 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:44.578751    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:45:44.579729 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.579703    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:45:44.579995 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.579979    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qlc8m\""
Sep 16 18:45:44.580256 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.580239    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:45:44.580537 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.580522    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.581971 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.581948    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.583983 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.583941    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:45:44.583983 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.583977    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:45:44.584669 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.584506    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:45:44.584669 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.584616    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:45:44.584834 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.584669    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9kn7w\""
Sep 16 18:45:44.584834 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.584509    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:45:44.585382 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.585071    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.585855 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.585833    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:45:44.585967 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.585833    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9krvl\""
Sep 16 18:45:44.588019 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.587997    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:44.588317 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.588298    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mp5mh\""
Sep 16 18:45:44.588503 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.588484    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:44.592995 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.592971    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:44.598117 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.598082    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ncwfs"
Sep 16 18:45:44.606254 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.606227    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ncwfs"
Sep 16 18:45:44.663098 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.663024    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:45:44.676035 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.675967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd6df1fa-d3f7-4f42-8476-4a1d8480e14d-host-slash\") pod \"iptables-alerter-jqjj6\" (UID: \"fd6df1fa-d3f7-4f42-8476-4a1d8480e14d\") " pod="openshift-network-operator/iptables-alerter-jqjj6"
Sep 16 18:45:44.676242 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-log-socket\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.676242 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676085    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61a115fb-6511-458e-9a4e-731f6e2bdfec-ovnkube-config\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.676242 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676113    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd6df1fa-d3f7-4f42-8476-4a1d8480e14d-host-slash\") pod \"iptables-alerter-jqjj6\" (UID: \"fd6df1fa-d3f7-4f42-8476-4a1d8480e14d\") " pod="openshift-network-operator/iptables-alerter-jqjj6"
Sep 16 18:45:44.676242 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676116    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61a115fb-6511-458e-9a4e-731f6e2bdfec-ovn-node-metrics-cert\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.676242 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ca622a1-571a-4769-a681-91f848125f1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.676242 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w47pm\" (UniqueName: \"kubernetes.io/projected/8ca622a1-571a-4769-a681-91f848125f1c-kube-api-access-w47pm\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.676591 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676247    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f0c80db-ca40-4eec-b140-a8e43262ed47-dbus\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:44.676591 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676292    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-var-lib-kubelet\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.676591 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9-agent-certs\") pod \"konnectivity-agent-glsm5\" (UID: \"6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9\") " pod="kube-system/konnectivity-agent-glsm5"
Sep 16 18:45:44.676591 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-run-netns\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.676591 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676467    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-cni-netd\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.676591 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676499    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.676591 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx2bd\" (UniqueName: \"kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd\") pod \"network-check-target-jsfns\" (UID: \"8339946f-2f59-4fce-93a4-a2a320b11588\") " pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:45:44.677043 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f0c80db-ca40-4eec-b140-a8e43262ed47-kubelet-config\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:44.677043 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676638    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7chp\" (UniqueName: \"kubernetes.io/projected/091a5344-37ad-4181-9ef5-018af0208dc7-kube-api-access-v7chp\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.677043 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676679    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f66f67-8c13-4880-9052-e0624f617592-tmp-dir\") pod \"node-resolver-stzn8\" (UID: \"36f66f67-8c13-4880-9052-e0624f617592\") " pod="openshift-dns/node-resolver-stzn8"
Sep 16 18:45:44.677043 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs\") pod \"network-metrics-daemon-2wlpl\" (UID: \"ae831f85-3302-4b52-8cea-c9f437a9409e\") " pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:44.677043 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-run\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.677043 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676790    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9-konnectivity-ca\") pod \"konnectivity-agent-glsm5\" (UID: \"6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9\") " pod="kube-system/konnectivity-agent-glsm5"
Sep 16 18:45:44.677043 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb9b2\" (UniqueName: \"kubernetes.io/projected/ae831f85-3302-4b52-8cea-c9f437a9409e-kube-api-access-tb9b2\") pod \"network-metrics-daemon-2wlpl\" (UID: \"ae831f85-3302-4b52-8cea-c9f437a9409e\") " pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:44.677043 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676850    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-run-netns\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.677043 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676877    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-var-lib-cni-bin\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.677043 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676919    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0480a6a4-52d4-4bd7-843d-63fff689d21d-tmp\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.677043 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.677043 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.676984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.677043 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677006    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-run-k8s-cni-cncf-io\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-kubelet\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-cni-bin\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61a115fb-6511-458e-9a4e-731f6e2bdfec-env-overrides\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61a115fb-6511-458e-9a4e-731f6e2bdfec-ovnkube-script-lib\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36f66f67-8c13-4880-9052-e0624f617592-hosts-file\") pod \"node-resolver-stzn8\" (UID: \"36f66f67-8c13-4880-9052-e0624f617592\") " pod="openshift-dns/node-resolver-stzn8"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677316    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-registration-dir\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677346    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/091a5344-37ad-4181-9ef5-018af0208dc7-cni-binary-copy\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-systemd-units\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677398    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzbzt\" (UniqueName: \"kubernetes.io/projected/61a115fb-6511-458e-9a4e-731f6e2bdfec-kube-api-access-dzbzt\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677465    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9-konnectivity-ca\") pod \"konnectivity-agent-glsm5\" (UID: \"6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9\") " pod="kube-system/konnectivity-agent-glsm5"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc24f\" (UniqueName: \"kubernetes.io/projected/36f66f67-8c13-4880-9052-e0624f617592-kube-api-access-xc24f\") pod \"node-resolver-stzn8\" (UID: \"36f66f67-8c13-4880-9052-e0624f617592\") " pod="openshift-dns/node-resolver-stzn8"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjlm5\" (UniqueName: \"kubernetes.io/projected/13f1c42c-201f-4df0-963f-b3a4165c2a68-kube-api-access-cjlm5\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677632    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-multus-socket-dir-parent\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677667    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-var-lib-kubelet\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.677773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677732    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-etc-kubernetes\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677761    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-sysconfig\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677788    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-kubernetes\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-sysctl-d\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfxb5\" (UniqueName: \"kubernetes.io/projected/fd6df1fa-d3f7-4f42-8476-4a1d8480e14d-kube-api-access-vfxb5\") pod \"iptables-alerter-jqjj6\" (UID: \"fd6df1fa-d3f7-4f42-8476-4a1d8480e14d\") " pod="openshift-network-operator/iptables-alerter-jqjj6"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677923    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-run-openvswitch\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677958    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shv24\" (UniqueName: \"kubernetes.io/projected/9b5b5c2b-18f2-4855-bb08-5173767444bf-kube-api-access-shv24\") pod \"node-ca-z9b6q\" (UID: \"9b5b5c2b-18f2-4855-bb08-5173767444bf\") " pod="openshift-image-registry/node-ca-z9b6q"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.677987    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ca622a1-571a-4769-a681-91f848125f1c-cnibin\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678040    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-socket-dir\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-system-cni-dir\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-multus-conf-dir\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-modprobe-d\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-run-systemd\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b5b5c2b-18f2-4855-bb08-5173767444bf-serviceca\") pod \"node-ca-z9b6q\" (UID: \"9b5b5c2b-18f2-4855-bb08-5173767444bf\") " pod="openshift-image-registry/node-ca-z9b6q"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8ca622a1-571a-4769-a681-91f848125f1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-systemd\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.678557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678276    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-host\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlrc6\" (UniqueName: \"kubernetes.io/projected/0480a6a4-52d4-4bd7-843d-63fff689d21d-kube-api-access-hlrc6\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678356    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-slash\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-multus-cni-dir\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678443    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-etc-openvswitch\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678487    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8ca622a1-571a-4769-a681-91f848125f1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678519    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-var-lib-cni-multus\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-hostroot\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678572    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-var-lib-openvswitch\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678616    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-run-ovn\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678642    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b5b5c2b-18f2-4855-bb08-5173767444bf-host\") pod \"node-ca-z9b6q\" (UID: \"9b5b5c2b-18f2-4855-bb08-5173767444bf\") " pod="openshift-image-registry/node-ca-z9b6q"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678670    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678712    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-device-dir\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-cnibin\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678791    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-sys\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678886    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-lib-modules\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.679480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678911    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-node-log\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.680226 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678950    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca622a1-571a-4769-a681-91f848125f1c-system-cni-dir\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.680226 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.678978    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ca622a1-571a-4769-a681-91f848125f1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.680226 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.679013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-sys-fs\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.680226 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.679041    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/091a5344-37ad-4181-9ef5-018af0208dc7-multus-daemon-config\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.680226 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.679119    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-run-multus-certs\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.680226 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.679146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-sysctl-conf\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.680226 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.679186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-tuned\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.680226 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.679229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd6df1fa-d3f7-4f42-8476-4a1d8480e14d-iptables-alerter-script\") pod \"iptables-alerter-jqjj6\" (UID: \"fd6df1fa-d3f7-4f42-8476-4a1d8480e14d\") " pod="openshift-network-operator/iptables-alerter-jqjj6"
Sep 16 18:45:44.681026 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.680964    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd6df1fa-d3f7-4f42-8476-4a1d8480e14d-iptables-alerter-script\") pod \"iptables-alerter-jqjj6\" (UID: \"fd6df1fa-d3f7-4f42-8476-4a1d8480e14d\") " pod="openshift-network-operator/iptables-alerter-jqjj6"
Sep 16 18:45:44.683625 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.683411    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9-agent-certs\") pod \"konnectivity-agent-glsm5\" (UID: \"6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9\") " pod="kube-system/konnectivity-agent-glsm5"
Sep 16 18:45:44.691821 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.691788    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfxb5\" (UniqueName: \"kubernetes.io/projected/fd6df1fa-d3f7-4f42-8476-4a1d8480e14d-kube-api-access-vfxb5\") pod \"iptables-alerter-jqjj6\" (UID: \"fd6df1fa-d3f7-4f42-8476-4a1d8480e14d\") " pod="openshift-network-operator/iptables-alerter-jqjj6"
Sep 16 18:45:44.703557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.703455    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-173.ec2.internal" event={"ID":"a86b7e4f585f41568b1e18c5a2fe21b4","Type":"ContainerStarted","Data":"f9713311e4dba1cab0acd91c049a864b69d78bdca6d7b32d388b3d33347996e9"}
Sep 16 18:45:44.706344 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.706300    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal" event={"ID":"2f14cf8cd09ed7a1368917ccea4b43bd","Type":"ContainerStarted","Data":"89286d1686d52a48905b2ebc606ffb15fb078d7712a93febc74ac9fbb1b2a0cf"}
Sep 16 18:45:44.779674 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.779620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-var-lib-openvswitch\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.779674 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.779666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-run-ovn\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.779912 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.779724    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-run-ovn\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.779912 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.779733    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-var-lib-openvswitch\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.779912 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.779798    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b5b5c2b-18f2-4855-bb08-5173767444bf-host\") pod \"node-ca-z9b6q\" (UID: \"9b5b5c2b-18f2-4855-bb08-5173767444bf\") " pod="openshift-image-registry/node-ca-z9b6q"
Sep 16 18:45:44.779912 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.779833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:44.779912 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.779858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-device-dir\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.779912 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.779881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-cnibin\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.779912 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.779891    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b5b5c2b-18f2-4855-bb08-5173767444bf-host\") pod \"node-ca-z9b6q\" (UID: \"9b5b5c2b-18f2-4855-bb08-5173767444bf\") " pod="openshift-image-registry/node-ca-z9b6q"
Sep 16 18:45:44.780264 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.779953    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-sys\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.780264 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.779896    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-sys\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.780264 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:44.779994    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:44.780264 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:44.780063    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret podName:5f0c80db-ca40-4eec-b140-a8e43262ed47 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.280042274 +0000 UTC m=+3.110650799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret") pod "global-pull-secret-syncer-bdq8w" (UID: "5f0c80db-ca40-4eec-b140-a8e43262ed47") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:44.780264 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780103    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-lib-modules\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.780264 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780158    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-cnibin\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.780264 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780202    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-device-dir\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.780264 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.779995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-lib-modules\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.780264 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780237    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-node-log\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.780264 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca622a1-571a-4769-a681-91f848125f1c-system-cni-dir\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ca622a1-571a-4769-a681-91f848125f1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780296    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-node-log\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780313    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-sys-fs\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780333    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca622a1-571a-4769-a681-91f848125f1c-system-cni-dir\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780350    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/091a5344-37ad-4181-9ef5-018af0208dc7-multus-daemon-config\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-run-multus-certs\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-sysctl-conf\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-tuned\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-log-socket\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61a115fb-6511-458e-9a4e-731f6e2bdfec-ovnkube-config\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780481    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ca622a1-571a-4769-a681-91f848125f1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780495    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61a115fb-6511-458e-9a4e-731f6e2bdfec-ovn-node-metrics-cert\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780518    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ca622a1-571a-4769-a681-91f848125f1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w47pm\" (UniqueName: \"kubernetes.io/projected/8ca622a1-571a-4769-a681-91f848125f1c-kube-api-access-w47pm\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780570    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f0c80db-ca40-4eec-b140-a8e43262ed47-dbus\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780595    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-var-lib-kubelet\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.780793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780599    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-sysctl-conf\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780621    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-run-netns\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780665    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-sys-fs\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-cni-netd\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780703    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx2bd\" (UniqueName: \"kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd\") pod \"network-check-target-jsfns\" (UID: \"8339946f-2f59-4fce-93a4-a2a320b11588\") " pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f0c80db-ca40-4eec-b140-a8e43262ed47-kubelet-config\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7chp\" (UniqueName: \"kubernetes.io/projected/091a5344-37ad-4181-9ef5-018af0208dc7-kube-api-access-v7chp\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f66f67-8c13-4880-9052-e0624f617592-tmp-dir\") pod \"node-resolver-stzn8\" (UID: \"36f66f67-8c13-4880-9052-e0624f617592\") " pod="openshift-dns/node-resolver-stzn8"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780834    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs\") pod \"network-metrics-daemon-2wlpl\" (UID: \"ae831f85-3302-4b52-8cea-c9f437a9409e\") " pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-run\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780888    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb9b2\" (UniqueName: \"kubernetes.io/projected/ae831f85-3302-4b52-8cea-c9f437a9409e-kube-api-access-tb9b2\") pod \"network-metrics-daemon-2wlpl\" (UID: \"ae831f85-3302-4b52-8cea-c9f437a9409e\") " pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-run-netns\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780944    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-var-lib-cni-bin\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0480a6a4-52d4-4bd7-843d-63fff689d21d-tmp\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.780996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.781682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-run-k8s-cni-cncf-io\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-kubelet\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781099    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-cni-bin\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781125    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61a115fb-6511-458e-9a4e-731f6e2bdfec-env-overrides\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781153    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61a115fb-6511-458e-9a4e-731f6e2bdfec-ovnkube-script-lib\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36f66f67-8c13-4880-9052-e0624f617592-hosts-file\") pod \"node-resolver-stzn8\" (UID: \"36f66f67-8c13-4880-9052-e0624f617592\") " pod="openshift-dns/node-resolver-stzn8"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-registration-dir\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781264    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/091a5344-37ad-4181-9ef5-018af0208dc7-cni-binary-copy\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-systemd-units\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzbzt\" (UniqueName: \"kubernetes.io/projected/61a115fb-6511-458e-9a4e-731f6e2bdfec-kube-api-access-dzbzt\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc24f\" (UniqueName: \"kubernetes.io/projected/36f66f67-8c13-4880-9052-e0624f617592-kube-api-access-xc24f\") pod \"node-resolver-stzn8\" (UID: \"36f66f67-8c13-4880-9052-e0624f617592\") " pod="openshift-dns/node-resolver-stzn8"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/091a5344-37ad-4181-9ef5-018af0208dc7-multus-daemon-config\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjlm5\" (UniqueName: \"kubernetes.io/projected/13f1c42c-201f-4df0-963f-b3a4165c2a68-kube-api-access-cjlm5\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-multus-socket-dir-parent\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781431    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-cni-netd\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-var-lib-kubelet\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.782548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-etc-kubernetes\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ca622a1-571a-4769-a681-91f848125f1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781512    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-sysconfig\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-kubernetes\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-sysctl-d\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781562    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-log-socket\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781580    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f0c80db-ca40-4eec-b140-a8e43262ed47-dbus\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-run-openvswitch\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shv24\" (UniqueName: \"kubernetes.io/projected/9b5b5c2b-18f2-4855-bb08-5173767444bf-kube-api-access-shv24\") pod \"node-ca-z9b6q\" (UID: \"9b5b5c2b-18f2-4855-bb08-5173767444bf\") " pod="openshift-image-registry/node-ca-z9b6q"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ca622a1-571a-4769-a681-91f848125f1c-cnibin\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-socket-dir\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781733    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-system-cni-dir\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781760    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-multus-conf-dir\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-modprobe-d\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781813    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-run-systemd\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b5b5c2b-18f2-4855-bb08-5173767444bf-serviceca\") pod \"node-ca-z9b6q\" (UID: \"9b5b5c2b-18f2-4855-bb08-5173767444bf\") " pod="openshift-image-registry/node-ca-z9b6q"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8ca622a1-571a-4769-a681-91f848125f1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.783434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781896    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-systemd\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-host\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781948    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlrc6\" (UniqueName: \"kubernetes.io/projected/0480a6a4-52d4-4bd7-843d-63fff689d21d-kube-api-access-hlrc6\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.781977    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-slash\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782001    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-multus-cni-dir\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782028    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-etc-openvswitch\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8ca622a1-571a-4769-a681-91f848125f1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-var-lib-cni-multus\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782110    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-sysconfig\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782129    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61a115fb-6511-458e-9a4e-731f6e2bdfec-ovnkube-config\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-hostroot\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782162    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-hostroot\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-run-systemd\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782351    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-etc-selinux\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782399    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-multus-socket-dir-parent\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782430    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-var-lib-kubelet\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782464    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-etc-kubernetes\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782743    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b5b5c2b-18f2-4855-bb08-5173767444bf-serviceca\") pod \"node-ca-z9b6q\" (UID: \"9b5b5c2b-18f2-4855-bb08-5173767444bf\") " pod="openshift-image-registry/node-ca-z9b6q"
Sep 16 18:45:44.784322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782781    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f66f67-8c13-4880-9052-e0624f617592-tmp-dir\") pod \"node-resolver-stzn8\" (UID: \"36f66f67-8c13-4880-9052-e0624f617592\") " pod="openshift-dns/node-resolver-stzn8"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.782869    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f0c80db-ca40-4eec-b140-a8e43262ed47-kubelet-config\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783012    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-run-multus-certs\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783060    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ca622a1-571a-4769-a681-91f848125f1c-cnibin\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783118    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-slash\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783151    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-registration-dir\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783170    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-systemd\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783212    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-host\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783256    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-kubernetes\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783292    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-run-ovn-kubernetes\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783338    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-var-lib-kubelet\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783450    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-run-openvswitch\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783450    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-sysctl-d\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-run-k8s-cni-cncf-io\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783501    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-run-netns\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.785261 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783576    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-kubelet\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783588    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-tuned\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783686    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-etc-openvswitch\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783759    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-multus-cni-dir\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:44.783804    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783843    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-run\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:44.783863    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs podName:ae831f85-3302-4b52-8cea-c9f437a9409e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.283844937 +0000 UTC m=+3.114453465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs") pod "network-metrics-daemon-2wlpl" (UID: "ae831f85-3302-4b52-8cea-c9f437a9409e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783904    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-run-netns\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.783914    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-var-lib-cni-bin\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.784199    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-system-cni-dir\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.784267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36f66f67-8c13-4880-9052-e0624f617592-hosts-file\") pod \"node-resolver-stzn8\" (UID: \"36f66f67-8c13-4880-9052-e0624f617592\") " pod="openshift-dns/node-resolver-stzn8"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.784311    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-systemd-units\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.784311    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8ca622a1-571a-4769-a681-91f848125f1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.784352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-multus-conf-dir\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.784371    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/091a5344-37ad-4181-9ef5-018af0208dc7-host-var-lib-cni-multus\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.784452    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13f1c42c-201f-4df0-963f-b3a4165c2a68-socket-dir\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.784489    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0480a6a4-52d4-4bd7-843d-63fff689d21d-etc-modprobe-d\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.784532    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61a115fb-6511-458e-9a4e-731f6e2bdfec-host-cni-bin\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.786135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.784573    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61a115fb-6511-458e-9a4e-731f6e2bdfec-ovnkube-script-lib\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.787041 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.784957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61a115fb-6511-458e-9a4e-731f6e2bdfec-env-overrides\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.787041 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.785107    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/091a5344-37ad-4181-9ef5-018af0208dc7-cni-binary-copy\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.787041 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.785627    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8ca622a1-571a-4769-a681-91f848125f1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.787041 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.785935    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61a115fb-6511-458e-9a4e-731f6e2bdfec-ovn-node-metrics-cert\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.787328 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.787304    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0480a6a4-52d4-4bd7-843d-63fff689d21d-tmp\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.810041 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.809940    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shv24\" (UniqueName: \"kubernetes.io/projected/9b5b5c2b-18f2-4855-bb08-5173767444bf-kube-api-access-shv24\") pod \"node-ca-z9b6q\" (UID: \"9b5b5c2b-18f2-4855-bb08-5173767444bf\") " pod="openshift-image-registry/node-ca-z9b6q"
Sep 16 18:45:44.810203 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.810086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb9b2\" (UniqueName: \"kubernetes.io/projected/ae831f85-3302-4b52-8cea-c9f437a9409e-kube-api-access-tb9b2\") pod \"network-metrics-daemon-2wlpl\" (UID: \"ae831f85-3302-4b52-8cea-c9f437a9409e\") " pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:44.812299 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.812269    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzbzt\" (UniqueName: \"kubernetes.io/projected/61a115fb-6511-458e-9a4e-731f6e2bdfec-kube-api-access-dzbzt\") pod \"ovnkube-node-dvn2p\" (UID: \"61a115fb-6511-458e-9a4e-731f6e2bdfec\") " pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.812449 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.812353    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc24f\" (UniqueName: \"kubernetes.io/projected/36f66f67-8c13-4880-9052-e0624f617592-kube-api-access-xc24f\") pod \"node-resolver-stzn8\" (UID: \"36f66f67-8c13-4880-9052-e0624f617592\") " pod="openshift-dns/node-resolver-stzn8"
Sep 16 18:45:44.812947 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.812921    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjlm5\" (UniqueName: \"kubernetes.io/projected/13f1c42c-201f-4df0-963f-b3a4165c2a68-kube-api-access-cjlm5\") pod \"aws-ebs-csi-driver-node-m9cv9\" (UID: \"13f1c42c-201f-4df0-963f-b3a4165c2a68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.813364 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.813341    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w47pm\" (UniqueName: \"kubernetes.io/projected/8ca622a1-571a-4769-a681-91f848125f1c-kube-api-access-w47pm\") pod \"multus-additional-cni-plugins-4twzx\" (UID: \"8ca622a1-571a-4769-a681-91f848125f1c\") " pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.813895 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:44.813876    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:44.813895 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:44.813897    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:44.814057 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:44.813913    2582 projected.go:196] Error preparing data for projected volume kube-api-access-mx2bd for pod openshift-network-diagnostics/network-check-target-jsfns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:44.814057 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:44.813995    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd podName:8339946f-2f59-4fce-93a4-a2a320b11588 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.313973712 +0000 UTC m=+3.144582257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mx2bd" (UniqueName: "kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd") pod "network-check-target-jsfns" (UID: "8339946f-2f59-4fce-93a4-a2a320b11588") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:44.814267 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.814245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlrc6\" (UniqueName: \"kubernetes.io/projected/0480a6a4-52d4-4bd7-843d-63fff689d21d-kube-api-access-hlrc6\") pod \"tuned-wtwdx\" (UID: \"0480a6a4-52d4-4bd7-843d-63fff689d21d\") " pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:44.814357 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.814332    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7chp\" (UniqueName: \"kubernetes.io/projected/091a5344-37ad-4181-9ef5-018af0208dc7-kube-api-access-v7chp\") pod \"multus-4tbdg\" (UID: \"091a5344-37ad-4181-9ef5-018af0208dc7\") " pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.874967 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.874912    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jqjj6"
Sep 16 18:45:44.886810 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.886783    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-glsm5"
Sep 16 18:45:44.897225 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.897202    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:44.898393 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.898365    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:45:44.903415 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.903389    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z9b6q"
Sep 16 18:45:44.912490 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.912464    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9"
Sep 16 18:45:44.922426 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.922390    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-stzn8"
Sep 16 18:45:44.930366 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.930346    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4twzx"
Sep 16 18:45:44.939363 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.939336    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4tbdg"
Sep 16 18:45:44.947241 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:44.947216    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wtwdx"
Sep 16 18:45:45.286278 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:45.286144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:45.286278 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:45.286260    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs\") pod \"network-metrics-daemon-2wlpl\" (UID: \"ae831f85-3302-4b52-8cea-c9f437a9409e\") " pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:45.286549 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:45.286375    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:45.286549 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:45.286384    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:45.286549 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:45.286477    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret podName:5f0c80db-ca40-4eec-b140-a8e43262ed47 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:46.286452482 +0000 UTC m=+4.117061028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret") pod "global-pull-secret-syncer-bdq8w" (UID: "5f0c80db-ca40-4eec-b140-a8e43262ed47") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:45.286549 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:45.286499    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs podName:ae831f85-3302-4b52-8cea-c9f437a9409e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:46.286489945 +0000 UTC m=+4.117098509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs") pod "network-metrics-daemon-2wlpl" (UID: "ae831f85-3302-4b52-8cea-c9f437a9409e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:45.387030 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:45.386976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx2bd\" (UniqueName: \"kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd\") pod \"network-check-target-jsfns\" (UID: \"8339946f-2f59-4fce-93a4-a2a320b11588\") " pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:45:45.387288 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:45.387189    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:45.387288 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:45.387216    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:45.387288 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:45.387227    2582 projected.go:196] Error preparing data for projected volume kube-api-access-mx2bd for pod openshift-network-diagnostics/network-check-target-jsfns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:45.387436 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:45.387303    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd podName:8339946f-2f59-4fce-93a4-a2a320b11588 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:46.387279752 +0000 UTC m=+4.217888279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mx2bd" (UniqueName: "kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd") pod "network-check-target-jsfns" (UID: "8339946f-2f59-4fce-93a4-a2a320b11588") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:45.610395 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:45.610291    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:45 +0000 UTC" deadline="2028-04-25 02:45:47.08659725 +0000 UTC"
Sep 16 18:45:45.610395 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:45.610327    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14072h0m1.476275916s"
Sep 16 18:45:45.698502 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:45.698458    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:45.698719 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:45.698680    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:45:45.896275 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:45.896235    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca622a1_571a_4769_a681_91f848125f1c.slice/crio-83aef2ac64c94031c8408bfc841cf133ad3ffa6c18603f52a555bb70f273beea WatchSource:0}: Error finding container 83aef2ac64c94031c8408bfc841cf133ad3ffa6c18603f52a555bb70f273beea: Status 404 returned error can't find the container with id 83aef2ac64c94031c8408bfc841cf133ad3ffa6c18603f52a555bb70f273beea
Sep 16 18:45:45.909769 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:45.909738    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a115fb_6511_458e_9a4e_731f6e2bdfec.slice/crio-2ea2f6049b06cfe95137a3e15b695f690fa963bc88d164de274c3ebcb67d222d WatchSource:0}: Error finding container 2ea2f6049b06cfe95137a3e15b695f690fa963bc88d164de274c3ebcb67d222d: Status 404 returned error can't find the container with id 2ea2f6049b06cfe95137a3e15b695f690fa963bc88d164de274c3ebcb67d222d
Sep 16 18:45:45.911321 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:45.911292    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f66f67_8c13_4880_9052_e0624f617592.slice/crio-2e9942c7481ca024fed97f8a9ef7cebb2cc20e28b1bfcc8905200424c2d54794 WatchSource:0}: Error finding container 2e9942c7481ca024fed97f8a9ef7cebb2cc20e28b1bfcc8905200424c2d54794: Status 404 returned error can't find the container with id 2e9942c7481ca024fed97f8a9ef7cebb2cc20e28b1bfcc8905200424c2d54794
Sep 16 18:45:45.912135 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:45.912110    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0480a6a4_52d4_4bd7_843d_63fff689d21d.slice/crio-24a83069ede2b08f78251667132fff4eea8ff7ec43a41d4fa4176906f866d53a WatchSource:0}: Error finding container 24a83069ede2b08f78251667132fff4eea8ff7ec43a41d4fa4176906f866d53a: Status 404 returned error can't find the container with id 24a83069ede2b08f78251667132fff4eea8ff7ec43a41d4fa4176906f866d53a
Sep 16 18:45:45.916077 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:45.916026    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4ca46b_f380_4a32_8163_5cdd8b6f9ec9.slice/crio-589342d5d7bb5973707ab248ac62606cd97f83f7b3ebf459629ba8ceacf75b06 WatchSource:0}: Error finding container 589342d5d7bb5973707ab248ac62606cd97f83f7b3ebf459629ba8ceacf75b06: Status 404 returned error can't find the container with id 589342d5d7bb5973707ab248ac62606cd97f83f7b3ebf459629ba8ceacf75b06
Sep 16 18:45:45.918162 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:45.918137    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f1c42c_201f_4df0_963f_b3a4165c2a68.slice/crio-4d38a895e416406de51a80bdd3dba9bcfda01f09bc493055750faaf7e81ddfed WatchSource:0}: Error finding container 4d38a895e416406de51a80bdd3dba9bcfda01f09bc493055750faaf7e81ddfed: Status 404 returned error can't find the container with id 4d38a895e416406de51a80bdd3dba9bcfda01f09bc493055750faaf7e81ddfed
Sep 16 18:45:45.919286 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:45.919261    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6df1fa_d3f7_4f42_8476_4a1d8480e14d.slice/crio-9bef5c49dbf9f260f55586d80e35846760056e9cc787cb9dc6b59e30584f013d WatchSource:0}: Error finding container 9bef5c49dbf9f260f55586d80e35846760056e9cc787cb9dc6b59e30584f013d: Status 404 returned error can't find the container with id 9bef5c49dbf9f260f55586d80e35846760056e9cc787cb9dc6b59e30584f013d
Sep 16 18:45:45.921623 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:45.921596    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5b5c2b_18f2_4855_bb08_5173767444bf.slice/crio-967c247cdb3a1785a4af48767f685353381ff3df2d1e281efbe2b8579eee0892 WatchSource:0}: Error finding container 967c247cdb3a1785a4af48767f685353381ff3df2d1e281efbe2b8579eee0892: Status 404 returned error can't find the container with id 967c247cdb3a1785a4af48767f685353381ff3df2d1e281efbe2b8579eee0892
Sep 16 18:45:45.922545 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:45:45.922503    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091a5344_37ad_4181_9ef5_018af0208dc7.slice/crio-5cb712326a97dc9614f7fffa8f60553d8be3666eff367e19362b779c5bc6db0c WatchSource:0}: Error finding container 5cb712326a97dc9614f7fffa8f60553d8be3666eff367e19362b779c5bc6db0c: Status 404 returned error can't find the container with id 5cb712326a97dc9614f7fffa8f60553d8be3666eff367e19362b779c5bc6db0c
Sep 16 18:45:46.296000 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.295945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:46.296227 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.296050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs\") pod \"network-metrics-daemon-2wlpl\" (UID: \"ae831f85-3302-4b52-8cea-c9f437a9409e\") " pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:46.296227 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:46.296217    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:46.296354 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:46.296305    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs podName:ae831f85-3302-4b52-8cea-c9f437a9409e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:48.296279151 +0000 UTC m=+6.126887692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs") pod "network-metrics-daemon-2wlpl" (UID: "ae831f85-3302-4b52-8cea-c9f437a9409e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:46.296406 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:46.296394    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:46.296456 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:46.296434    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret podName:5f0c80db-ca40-4eec-b140-a8e43262ed47 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:48.296421408 +0000 UTC m=+6.127029935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret") pod "global-pull-secret-syncer-bdq8w" (UID: "5f0c80db-ca40-4eec-b140-a8e43262ed47") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:46.397215 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.397150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx2bd\" (UniqueName: \"kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd\") pod \"network-check-target-jsfns\" (UID: \"8339946f-2f59-4fce-93a4-a2a320b11588\") " pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:45:46.397470 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:46.397446    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:46.397574 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:46.397507    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:46.397574 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:46.397541    2582 projected.go:196] Error preparing data for projected volume kube-api-access-mx2bd for pod openshift-network-diagnostics/network-check-target-jsfns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:46.397726 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:46.397620    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd podName:8339946f-2f59-4fce-93a4-a2a320b11588 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:48.397597413 +0000 UTC m=+6.228205938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mx2bd" (UniqueName: "kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd") pod "network-check-target-jsfns" (UID: "8339946f-2f59-4fce-93a4-a2a320b11588") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:46.611235 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.611085    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:45 +0000 UTC" deadline="2028-06-17 06:56:41.535013064 +0000 UTC"
Sep 16 18:45:46.611235 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.611169    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15348h10m54.923849561s"
Sep 16 18:45:46.698880 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.698832    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:46.699086 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:46.699014    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:45:46.699895 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.699688    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:45:46.699895 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:46.699839    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:45:46.723588 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.723531    2582 generic.go:356] "Generic (PLEG): container finished" podID="2f14cf8cd09ed7a1368917ccea4b43bd" containerID="637f6bcde28ad84b0dbd9532771eed1e2dfa0b9324824027645d9d5f57b33336" exitCode=0
Sep 16 18:45:46.724457 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.724422    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal" event={"ID":"2f14cf8cd09ed7a1368917ccea4b43bd","Type":"ContainerDied","Data":"637f6bcde28ad84b0dbd9532771eed1e2dfa0b9324824027645d9d5f57b33336"}
Sep 16 18:45:46.733066 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.733027    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9" event={"ID":"13f1c42c-201f-4df0-963f-b3a4165c2a68","Type":"ContainerStarted","Data":"4d38a895e416406de51a80bdd3dba9bcfda01f09bc493055750faaf7e81ddfed"}
Sep 16 18:45:46.745023 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.744980    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p" event={"ID":"61a115fb-6511-458e-9a4e-731f6e2bdfec","Type":"ContainerStarted","Data":"2ea2f6049b06cfe95137a3e15b695f690fa963bc88d164de274c3ebcb67d222d"}
Sep 16 18:45:46.752949 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.752909    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wtwdx" event={"ID":"0480a6a4-52d4-4bd7-843d-63fff689d21d","Type":"ContainerStarted","Data":"24a83069ede2b08f78251667132fff4eea8ff7ec43a41d4fa4176906f866d53a"}
Sep 16 18:45:46.767546 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.767493    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-stzn8" event={"ID":"36f66f67-8c13-4880-9052-e0624f617592","Type":"ContainerStarted","Data":"2e9942c7481ca024fed97f8a9ef7cebb2cc20e28b1bfcc8905200424c2d54794"}
Sep 16 18:45:46.786589 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.786531    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4twzx" event={"ID":"8ca622a1-571a-4769-a681-91f848125f1c","Type":"ContainerStarted","Data":"83aef2ac64c94031c8408bfc841cf133ad3ffa6c18603f52a555bb70f273beea"}
Sep 16 18:45:46.798201 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.798137    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-173.ec2.internal" event={"ID":"a86b7e4f585f41568b1e18c5a2fe21b4","Type":"ContainerStarted","Data":"eb36d5e67e04602ad2ad1a8999e7377fb17f2f5c9c1254bd89395868506395e5"}
Sep 16 18:45:46.812298 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.811976    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jqjj6" event={"ID":"fd6df1fa-d3f7-4f42-8476-4a1d8480e14d","Type":"ContainerStarted","Data":"9bef5c49dbf9f260f55586d80e35846760056e9cc787cb9dc6b59e30584f013d"}
Sep 16 18:45:46.816676 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.813904    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4tbdg" event={"ID":"091a5344-37ad-4181-9ef5-018af0208dc7","Type":"ContainerStarted","Data":"5cb712326a97dc9614f7fffa8f60553d8be3666eff367e19362b779c5bc6db0c"}
Sep 16 18:45:46.833134 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.833077    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z9b6q" event={"ID":"9b5b5c2b-18f2-4855-bb08-5173767444bf","Type":"ContainerStarted","Data":"967c247cdb3a1785a4af48767f685353381ff3df2d1e281efbe2b8579eee0892"}
Sep 16 18:45:46.840332 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:46.840232    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-glsm5" event={"ID":"6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9","Type":"ContainerStarted","Data":"589342d5d7bb5973707ab248ac62606cd97f83f7b3ebf459629ba8ceacf75b06"}
Sep 16 18:45:47.698422 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:47.697839    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:47.698422 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:47.698007    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:45:47.867539 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:47.867493    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal" event={"ID":"2f14cf8cd09ed7a1368917ccea4b43bd","Type":"ContainerStarted","Data":"1d1740b648c58fbbb2e14703f75c5517bcebb8673c16619a07d513c176a7b52c"}
Sep 16 18:45:47.885520 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:47.885370    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-173.ec2.internal" podStartSLOduration=3.885342307 podStartE2EDuration="3.885342307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:46.816140109 +0000 UTC m=+4.646748657" watchObservedRunningTime="2026-09-16 18:45:47.885342307 +0000 UTC m=+5.715950856"
Sep 16 18:45:48.317437 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:48.317401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs\") pod \"network-metrics-daemon-2wlpl\" (UID: \"ae831f85-3302-4b52-8cea-c9f437a9409e\") " pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:48.317628 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:48.317483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:48.317740 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:48.317658    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:48.317740 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:48.317702    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:48.317740 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:48.317734    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret podName:5f0c80db-ca40-4eec-b140-a8e43262ed47 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:52.317708645 +0000 UTC m=+10.148317194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret") pod "global-pull-secret-syncer-bdq8w" (UID: "5f0c80db-ca40-4eec-b140-a8e43262ed47") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:48.317970 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:48.317752    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs podName:ae831f85-3302-4b52-8cea-c9f437a9409e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:52.317743501 +0000 UTC m=+10.148352026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs") pod "network-metrics-daemon-2wlpl" (UID: "ae831f85-3302-4b52-8cea-c9f437a9409e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:48.418041 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:48.417990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx2bd\" (UniqueName: \"kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd\") pod \"network-check-target-jsfns\" (UID: \"8339946f-2f59-4fce-93a4-a2a320b11588\") " pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:45:48.418234 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:48.418212    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:48.418234 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:48.418234    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:48.418367 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:48.418247    2582 projected.go:196] Error preparing data for projected volume kube-api-access-mx2bd for pod openshift-network-diagnostics/network-check-target-jsfns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:48.418367 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:48.418309    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd podName:8339946f-2f59-4fce-93a4-a2a320b11588 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:52.418288641 +0000 UTC m=+10.248897169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mx2bd" (UniqueName: "kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd") pod "network-check-target-jsfns" (UID: "8339946f-2f59-4fce-93a4-a2a320b11588") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:48.698816 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:48.698736    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:45:48.699280 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:48.698872    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:45:48.699343 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:48.699284    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:48.699449 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:48.699389    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:45:49.698276 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:49.698237    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:49.698513 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:49.698396    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:45:50.700953 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:50.700204    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:45:50.700953 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:50.700344    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:45:50.700953 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:50.700789    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:50.700953 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:50.700892    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:45:51.698526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:51.698398    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:51.698758 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:51.698632    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:45:52.353434 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:52.353388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs\") pod \"network-metrics-daemon-2wlpl\" (UID: \"ae831f85-3302-4b52-8cea-c9f437a9409e\") " pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:52.353912 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:52.353479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:52.353912 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:52.353650    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:52.353912 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:52.353727    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret podName:5f0c80db-ca40-4eec-b140-a8e43262ed47 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:00.353706154 +0000 UTC m=+18.184314682 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret") pod "global-pull-secret-syncer-bdq8w" (UID: "5f0c80db-ca40-4eec-b140-a8e43262ed47") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:52.354178 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:52.354157    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:52.354267 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:52.354219    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs podName:ae831f85-3302-4b52-8cea-c9f437a9409e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:00.354203063 +0000 UTC m=+18.184811603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs") pod "network-metrics-daemon-2wlpl" (UID: "ae831f85-3302-4b52-8cea-c9f437a9409e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:52.454421 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:52.454378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx2bd\" (UniqueName: \"kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd\") pod \"network-check-target-jsfns\" (UID: \"8339946f-2f59-4fce-93a4-a2a320b11588\") " pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:45:52.454646 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:52.454627    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:52.454711 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:52.454652    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:52.454711 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:52.454666    2582 projected.go:196] Error preparing data for projected volume kube-api-access-mx2bd for pod openshift-network-diagnostics/network-check-target-jsfns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:52.454807 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:52.454737    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd podName:8339946f-2f59-4fce-93a4-a2a320b11588 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:00.454717238 +0000 UTC m=+18.285325762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mx2bd" (UniqueName: "kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd") pod "network-check-target-jsfns" (UID: "8339946f-2f59-4fce-93a4-a2a320b11588") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:52.699884 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:52.699790    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:52.700053 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:52.699915    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:45:52.700306 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:52.700288    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:45:52.700435 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:52.700381    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:45:53.698049 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:53.698002    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:53.698598 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:53.698204    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:45:54.698645 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:54.698602    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:54.699069 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:54.698599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:45:54.699069 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:54.698776    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:45:54.699069 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:54.698805    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:45:55.697857 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:55.697816    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:55.698085 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:55.698001    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:45:56.698401 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:56.698347    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:45:56.698401 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:56.698400    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:56.698952 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:56.698501    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:45:56.698952 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:56.698653    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:45:57.698537 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:57.698442    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:57.699048 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:57.698596    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:45:58.698365 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:58.698329    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:45:58.698556 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:58.698375    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:45:58.698556 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:58.698455    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:45:58.698968 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:58.698603    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:45:59.698162 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:45:59.698124    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:45:59.698435 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:45:59.698244    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:46:00.417751 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:00.417705    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs\") pod \"network-metrics-daemon-2wlpl\" (UID: \"ae831f85-3302-4b52-8cea-c9f437a9409e\") " pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:00.418290 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:00.417783    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:00.418290 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:00.417876    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:46:00.418290 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:00.417876    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:46:00.418290 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:00.417947    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs podName:ae831f85-3302-4b52-8cea-c9f437a9409e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:16.417930752 +0000 UTC m=+34.248539274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs") pod "network-metrics-daemon-2wlpl" (UID: "ae831f85-3302-4b52-8cea-c9f437a9409e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:46:00.418290 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:00.417962    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret podName:5f0c80db-ca40-4eec-b140-a8e43262ed47 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:16.417956682 +0000 UTC m=+34.248565205 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret") pod "global-pull-secret-syncer-bdq8w" (UID: "5f0c80db-ca40-4eec-b140-a8e43262ed47") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:46:00.518943 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:00.518900    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx2bd\" (UniqueName: \"kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd\") pod \"network-check-target-jsfns\" (UID: \"8339946f-2f59-4fce-93a4-a2a320b11588\") " pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:00.519120 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:00.519079    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:46:00.519120 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:00.519112    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:46:00.519207 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:00.519128    2582 projected.go:196] Error preparing data for projected volume kube-api-access-mx2bd for pod openshift-network-diagnostics/network-check-target-jsfns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:46:00.519207 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:00.519201    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd podName:8339946f-2f59-4fce-93a4-a2a320b11588 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:16.519179801 +0000 UTC m=+34.349788325 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mx2bd" (UniqueName: "kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd") pod "network-check-target-jsfns" (UID: "8339946f-2f59-4fce-93a4-a2a320b11588") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:46:00.698289 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:00.698191    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:00.698289 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:00.698251    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:00.698515 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:00.698349    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:46:00.698515 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:00.698475    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:46:01.697868 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:01.697829    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:01.698296 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:01.697988    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:46:02.699080 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:02.699039    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:02.699515 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:02.699165    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:46:02.699681 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:02.699652    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:02.699817 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:02.699761    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:46:03.698438 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:03.698398    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:03.698630 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:03.698521    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:46:04.701698 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.701528    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:04.701698 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.701568    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:04.702224 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:04.701701    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:46:04.702224 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:04.701834    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:46:04.909208 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.908981    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9" event={"ID":"13f1c42c-201f-4df0-963f-b3a4165c2a68","Type":"ContainerStarted","Data":"ece702542f96fd7c9c3459c1b649ce17084b684c369186b2015d08d315189db2"}
Sep 16 18:46:04.911606 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.911555    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p" event={"ID":"61a115fb-6511-458e-9a4e-731f6e2bdfec","Type":"ContainerStarted","Data":"e63c2ef899bf80140713cb95a92978ad828d5048f649d5c4c47604c631f0f715"}
Sep 16 18:46:04.911713 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.911645    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p" event={"ID":"61a115fb-6511-458e-9a4e-731f6e2bdfec","Type":"ContainerStarted","Data":"3594c30c29e2453af7e7ede4e274c280953bb3e1b2c5be798aafbcc29937f65c"}
Sep 16 18:46:04.911713 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.911678    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p" event={"ID":"61a115fb-6511-458e-9a4e-731f6e2bdfec","Type":"ContainerStarted","Data":"87d433716018728237ebfa4a0ae7adbfa5f88b061c50857090f029c2e4973d98"}
Sep 16 18:46:04.913419 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.913394    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wtwdx" event={"ID":"0480a6a4-52d4-4bd7-843d-63fff689d21d","Type":"ContainerStarted","Data":"633a79710004757f40f795530ce2b64815847727f19c12265f7d7d1f2a32667c"}
Sep 16 18:46:04.915803 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.915774    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-stzn8" event={"ID":"36f66f67-8c13-4880-9052-e0624f617592","Type":"ContainerStarted","Data":"fb9fa1410d0083230a536acdc219dc03e414463dc8e5c127e52e0ec6d53066dd"}
Sep 16 18:46:04.921262 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.921220    2582 generic.go:356] "Generic (PLEG): container finished" podID="8ca622a1-571a-4769-a681-91f848125f1c" containerID="8dd39f015390aa3a47caebadc6b161c0d11ac10f2a8ceb7feba92689919a890c" exitCode=0
Sep 16 18:46:04.921356 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.921302    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4twzx" event={"ID":"8ca622a1-571a-4769-a681-91f848125f1c","Type":"ContainerDied","Data":"8dd39f015390aa3a47caebadc6b161c0d11ac10f2a8ceb7feba92689919a890c"}
Sep 16 18:46:04.924482 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.924456    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4tbdg" event={"ID":"091a5344-37ad-4181-9ef5-018af0208dc7","Type":"ContainerStarted","Data":"e27d62ac9cb7eb55f754576927bdd4b69aa37fb70f98128d7cbbe8f2d1fd6fe1"}
Sep 16 18:46:04.926236 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.926213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z9b6q" event={"ID":"9b5b5c2b-18f2-4855-bb08-5173767444bf","Type":"ContainerStarted","Data":"ec2f59cdfe2065a55340b0ba417ce7cabd9b28cf397a1b4230d9f18cad02891f"}
Sep 16 18:46:04.927918 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.927890    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-glsm5" event={"ID":"6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9","Type":"ContainerStarted","Data":"35b276573abdc34662906527a3966d457821fc2dca2a73a9c62c020a932a743a"}
Sep 16 18:46:04.930724 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.930690    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wtwdx" podStartSLOduration=3.281293508 podStartE2EDuration="21.930679637s" totalImagesPullingTime="18.649386129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:04.930510722 +0000 UTC m=+22.761119266" watchObservedRunningTime="2026-09-16 18:46:04.930679637 +0000 UTC m=+22.761288182"
Sep 16 18:46:04.931157 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.931137    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-173.ec2.internal" podStartSLOduration=20.931129963 podStartE2EDuration="20.931129963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:47.886803479 +0000 UTC m=+5.717412029" watchObservedRunningTime="2026-09-16 18:46:04.931129963 +0000 UTC m=+22.761738507"
Sep 16 18:46:04.964448 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.964249    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-glsm5" podStartSLOduration=3.360046255 podStartE2EDuration="21.964228754s" totalImagesPullingTime="18.604182499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:04.963948102 +0000 UTC m=+22.794556658" watchObservedRunningTime="2026-09-16 18:46:04.964228754 +0000 UTC m=+22.794837300"
Sep 16 18:46:04.997312 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.997271    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-stzn8" podStartSLOduration=3.359879226 podStartE2EDuration="21.997251551s" totalImagesPullingTime="18.637372325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:04.997214267 +0000 UTC m=+22.827822824" watchObservedRunningTime="2026-09-16 18:46:04.997251551 +0000 UTC m=+22.827860096"
Sep 16 18:46:04.997420 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:04.997344    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z9b6q" podStartSLOduration=3.398453013 podStartE2EDuration="21.99734056s" totalImagesPullingTime="18.598887547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:04.979183929 +0000 UTC m=+22.809792475" watchObservedRunningTime="2026-09-16 18:46:04.99734056 +0000 UTC m=+22.827949131"
Sep 16 18:46:05.081979 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:05.081946    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:05.154463 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:05.154427    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:05.210152 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:05.210117    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:05.697913 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:05.697703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:05.698053 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:05.698012    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:46:05.935324 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:05.935282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p" event={"ID":"61a115fb-6511-458e-9a4e-731f6e2bdfec","Type":"ContainerStarted","Data":"53210e4a0796f409f59dfe63822783364a96f06c47db531b5c64ad5c4682f3a2"}
Sep 16 18:46:05.936100 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:05.935382    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p" event={"ID":"61a115fb-6511-458e-9a4e-731f6e2bdfec","Type":"ContainerStarted","Data":"c8eb03cf770e59b73d3b234eab7befff5de87789d57e95c47a453ac3e93428a8"}
Sep 16 18:46:05.936100 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:05.935434    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p" event={"ID":"61a115fb-6511-458e-9a4e-731f6e2bdfec","Type":"ContainerStarted","Data":"629a273730aa8e23e9b42be2ad34bd3d2778ce093c34fe1fa1917890067a30a5"}
Sep 16 18:46:06.047920 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:06.047684    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:46:06.593955 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:06.593657    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-glsm5"
Sep 16 18:46:06.594361 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:06.594343    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-glsm5"
Sep 16 18:46:06.613545 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:06.613499    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4tbdg" podStartSLOduration=4.922473024 podStartE2EDuration="23.613486749s" totalImagesPullingTime="18.691013725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:05.017069357 +0000 UTC m=+22.847677902" watchObservedRunningTime="2026-09-16 18:46:06.613486749 +0000 UTC m=+24.444095291"
Sep 16 18:46:06.651104 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:06.651018    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:46:06.047706426Z","UUID":"e7c6d6c5-fdea-4a2b-9c69-9ee60099b054","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:46:06.652847 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:06.652823    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:46:06.652967 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:06.652856    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:46:06.698607 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:06.698508    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:06.698760 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:06.698645    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:46:06.698760 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:06.698717    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:06.698880 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:06.698818    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:46:06.940561 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:06.940325    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jqjj6" event={"ID":"fd6df1fa-d3f7-4f42-8476-4a1d8480e14d","Type":"ContainerStarted","Data":"bc96667b21f5c37aafcd4d0f439eb5e214eadffac9eab4a49b7a575b6263a647"}
Sep 16 18:46:06.942496 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:06.942466    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9" event={"ID":"13f1c42c-201f-4df0-963f-b3a4165c2a68","Type":"ContainerStarted","Data":"bcd2c01a8d4ede36d9a9d5aaad2ba301ed2ad76c85e062a21bf5be9f82ef9742"}
Sep 16 18:46:07.249830 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:07.249806    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:07.350795 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:07.350771    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:07.697844 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:07.697817    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:07.697996 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:07.697947    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:46:07.947208 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:07.947168    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9" event={"ID":"13f1c42c-201f-4df0-963f-b3a4165c2a68","Type":"ContainerStarted","Data":"714e9b54a9074d0dbfdc9d31c6567732371cf6ff48de1bda9db6e70c683e5847"}
Sep 16 18:46:07.951348 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:07.951288    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p" event={"ID":"61a115fb-6511-458e-9a4e-731f6e2bdfec","Type":"ContainerStarted","Data":"8b3b08b4e1eb27816915a1b56e23b151f280bda6e63369e9cff9ecfd4512e879"}
Sep 16 18:46:07.951348 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:07.951311    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:46:07.965603 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:07.965012    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m9cv9" podStartSLOduration=4.233052429 podStartE2EDuration="24.964999106s" totalImagesPullingTime="20.731946677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:07.964855449 +0000 UTC m=+25.795464019" watchObservedRunningTime="2026-09-16 18:46:07.964999106 +0000 UTC m=+25.795607642"
Sep 16 18:46:07.965603 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:07.965308    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jqjj6" podStartSLOduration=6.329748624 podStartE2EDuration="24.965299448s" totalImagesPullingTime="18.635550824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:06.956875054 +0000 UTC m=+24.787483598" watchObservedRunningTime="2026-09-16 18:46:07.965299448 +0000 UTC m=+25.795908009"
Sep 16 18:46:08.698668 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:08.698205    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:08.698668 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:08.698335    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:46:08.698668 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:08.698392    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:08.698668 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:08.698478    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:46:09.697790 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:09.697653    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:09.698226 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:09.697937    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:46:09.958874 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:09.958803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p" event={"ID":"61a115fb-6511-458e-9a4e-731f6e2bdfec","Type":"ContainerStarted","Data":"60a8c6e907dee6b5d253b33e9f466355f26f4fb1ece74d14de18354ce869e66e"}
Sep 16 18:46:09.959076 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:09.959062    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:46:09.959194 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:09.959124    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:46:09.960797 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:09.960772    2582 generic.go:356] "Generic (PLEG): container finished" podID="8ca622a1-571a-4769-a681-91f848125f1c" containerID="4d8f350f3c2ac92c50838ad9abdbbf36249aa1b919716e2d1e8e47715d2eb37f" exitCode=0
Sep 16 18:46:09.960892 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:09.960813    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4twzx" event={"ID":"8ca622a1-571a-4769-a681-91f848125f1c","Type":"ContainerDied","Data":"4d8f350f3c2ac92c50838ad9abdbbf36249aa1b919716e2d1e8e47715d2eb37f"}
Sep 16 18:46:09.974396 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:09.974373    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:46:09.986665 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:09.986629    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p" podStartSLOduration=8.284784006 podStartE2EDuration="26.986615767s" totalImagesPullingTime="18.701831761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:09.985189759 +0000 UTC m=+27.815798304" watchObservedRunningTime="2026-09-16 18:46:09.986615767 +0000 UTC m=+27.817224293"
Sep 16 18:46:10.698196 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:10.698169    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:10.698702 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:10.698285    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:46:10.698702 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:10.698348    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:10.698702 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:10.698440    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:46:10.966057 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:10.965981    2582 generic.go:356] "Generic (PLEG): container finished" podID="8ca622a1-571a-4769-a681-91f848125f1c" containerID="554693b46025af9294a85156b0ae5f548115eced06f855d746cf211147ce7161" exitCode=0
Sep 16 18:46:10.966211 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:10.966069    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4twzx" event={"ID":"8ca622a1-571a-4769-a681-91f848125f1c","Type":"ContainerDied","Data":"554693b46025af9294a85156b0ae5f548115eced06f855d746cf211147ce7161"}
Sep 16 18:46:10.966503 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:10.966476    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:46:10.981763 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:10.981740    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:46:11.304629 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:11.304422    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-glsm5"
Sep 16 18:46:11.304771 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:11.304757    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:46:11.304989 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:11.304971    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-glsm5"
Sep 16 18:46:11.372520 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:11.372494    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:11.466186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:11.465896    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bdq8w"]
Sep 16 18:46:11.466186 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:11.465992    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:11.466186 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:11.466069    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:46:11.469501 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:11.469481    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2wlpl"]
Sep 16 18:46:11.469643 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:11.469615    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:11.469731 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:11.469705    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:46:11.470270 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:11.470249    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jsfns"]
Sep 16 18:46:11.470352 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:11.470339    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:11.470425 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:11.470406    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:46:11.970257 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:11.970183    2582 generic.go:356] "Generic (PLEG): container finished" podID="8ca622a1-571a-4769-a681-91f848125f1c" containerID="e4f1dc6162b316c475a6039c950ac70b507c14ec352d51eb77104d2fdbb1927e" exitCode=0
Sep 16 18:46:11.970646 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:11.970271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4twzx" event={"ID":"8ca622a1-571a-4769-a681-91f848125f1c","Type":"ContainerDied","Data":"e4f1dc6162b316c475a6039c950ac70b507c14ec352d51eb77104d2fdbb1927e"}
Sep 16 18:46:12.698704 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:12.698678    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:12.698864 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:12.698765    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:12.698864 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:12.698767    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:46:12.698864 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:12.698854    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:46:12.699001 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:12.698892    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:12.699001 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:12.698959    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:46:14.697907 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:14.697874    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:14.698417 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:14.697875    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:14.698417 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:14.698005    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:14.698417 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:14.698008    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:46:14.698417 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:14.698110    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:46:14.698417 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:14.698214    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:46:16.439564 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:16.439516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:16.440022 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:16.439615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs\") pod \"network-metrics-daemon-2wlpl\" (UID: \"ae831f85-3302-4b52-8cea-c9f437a9409e\") " pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:16.440022 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:16.439690    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:46:16.440022 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:16.439737    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:46:16.440022 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:16.439767    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret podName:5f0c80db-ca40-4eec-b140-a8e43262ed47 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:48.439748479 +0000 UTC m=+66.270357006 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret") pod "global-pull-secret-syncer-bdq8w" (UID: "5f0c80db-ca40-4eec-b140-a8e43262ed47") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:46:16.440022 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:16.439791    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs podName:ae831f85-3302-4b52-8cea-c9f437a9409e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:48.439773624 +0000 UTC m=+66.270382162 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs") pod "network-metrics-daemon-2wlpl" (UID: "ae831f85-3302-4b52-8cea-c9f437a9409e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:46:16.540245 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:16.540161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx2bd\" (UniqueName: \"kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd\") pod \"network-check-target-jsfns\" (UID: \"8339946f-2f59-4fce-93a4-a2a320b11588\") " pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:16.540429 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:16.540336    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:46:16.540429 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:16.540363    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:46:16.540429 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:16.540373    2582 projected.go:196] Error preparing data for projected volume kube-api-access-mx2bd for pod openshift-network-diagnostics/network-check-target-jsfns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:46:16.540429 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:16.540429    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd podName:8339946f-2f59-4fce-93a4-a2a320b11588 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:48.540414226 +0000 UTC m=+66.371022776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mx2bd" (UniqueName: "kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd") pod "network-check-target-jsfns" (UID: "8339946f-2f59-4fce-93a4-a2a320b11588") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:46:16.698163 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:16.698130    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:16.698381 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:16.698170    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:16.698381 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:16.698254    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jsfns" podUID="8339946f-2f59-4fce-93a4-a2a320b11588"
Sep 16 18:46:16.698381 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:16.698294    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdq8w" podUID="5f0c80db-ca40-4eec-b140-a8e43262ed47"
Sep 16 18:46:16.698381 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:16.698308    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:16.698596 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:16.698394    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wlpl" podUID="ae831f85-3302-4b52-8cea-c9f437a9409e"
Sep 16 18:46:16.703240 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:16.703181    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:16.768598 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:16.768542    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:16.986281 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:16.986184    2582 generic.go:356] "Generic (PLEG): container finished" podID="8ca622a1-571a-4769-a681-91f848125f1c" containerID="029b78a3f5eed89bc99308d482895cf954f980c9b1cdd9742109b9e27f3c9a9c" exitCode=0
Sep 16 18:46:16.986281 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:16.986239    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4twzx" event={"ID":"8ca622a1-571a-4769-a681-91f848125f1c","Type":"ContainerDied","Data":"029b78a3f5eed89bc99308d482895cf954f980c9b1cdd9742109b9e27f3c9a9c"}
Sep 16 18:46:17.398641 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.398613    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-173.ec2.internal" event="NodeReady"
Sep 16 18:46:17.398809 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.398739    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:46:17.447857 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.447646    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tqdhb"]
Sep 16 18:46:17.478606 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.478551    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tnknt"]
Sep 16 18:46:17.478792 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.478683    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:17.481929 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.481907    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:46:17.482053 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.481949    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:46:17.482053 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.482024    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vf4dn\""
Sep 16 18:46:17.502502 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.502469    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tnknt"]
Sep 16 18:46:17.502502 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.502505    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tqdhb"]
Sep 16 18:46:17.502717 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.502620    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tnknt"
Sep 16 18:46:17.505716 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.505677    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:46:17.505868 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.505747    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s4v9d\""
Sep 16 18:46:17.505868 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.505808    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:46:17.505868 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.505808    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:46:17.572894 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.572858    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:17.648711 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.648625    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:17.648711 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.648657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz6nt\" (UniqueName: \"kubernetes.io/projected/7df2f4be-0543-439c-8a7a-2235ad96dc42-kube-api-access-nz6nt\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:17.648711 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.648676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7df2f4be-0543-439c-8a7a-2235ad96dc42-config-volume\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:17.648711 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.648696    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df2f4be-0543-439c-8a7a-2235ad96dc42-tmp-dir\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:17.648989 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.648719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf57ca8b-8820-478b-958a-b671cb621908-cert\") pod \"ingress-canary-tnknt\" (UID: \"bf57ca8b-8820-478b-958a-b671cb621908\") " pod="openshift-ingress-canary/ingress-canary-tnknt"
Sep 16 18:46:17.648989 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.648795    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5dkq\" (UniqueName: \"kubernetes.io/projected/bf57ca8b-8820-478b-958a-b671cb621908-kube-api-access-b5dkq\") pod \"ingress-canary-tnknt\" (UID: \"bf57ca8b-8820-478b-958a-b671cb621908\") " pod="openshift-ingress-canary/ingress-canary-tnknt"
Sep 16 18:46:17.674213 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.674186    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:17.749419 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.749380    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df2f4be-0543-439c-8a7a-2235ad96dc42-tmp-dir\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:17.749419 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.749416    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf57ca8b-8820-478b-958a-b671cb621908-cert\") pod \"ingress-canary-tnknt\" (UID: \"bf57ca8b-8820-478b-958a-b671cb621908\") " pod="openshift-ingress-canary/ingress-canary-tnknt"
Sep 16 18:46:17.749688 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.749449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5dkq\" (UniqueName: \"kubernetes.io/projected/bf57ca8b-8820-478b-958a-b671cb621908-kube-api-access-b5dkq\") pod \"ingress-canary-tnknt\" (UID: \"bf57ca8b-8820-478b-958a-b671cb621908\") " pod="openshift-ingress-canary/ingress-canary-tnknt"
Sep 16 18:46:17.749688 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.749501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:17.749688 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.749518    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz6nt\" (UniqueName: \"kubernetes.io/projected/7df2f4be-0543-439c-8a7a-2235ad96dc42-kube-api-access-nz6nt\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:17.749688 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.749535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7df2f4be-0543-439c-8a7a-2235ad96dc42-config-volume\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:17.749688 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:17.749655    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:17.749899 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:17.749742    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls podName:7df2f4be-0543-439c-8a7a-2235ad96dc42 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.249720081 +0000 UTC m=+36.080328607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls") pod "dns-default-tqdhb" (UID: "7df2f4be-0543-439c-8a7a-2235ad96dc42") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:17.750077 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.750054    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df2f4be-0543-439c-8a7a-2235ad96dc42-tmp-dir\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:17.750291 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.750272    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7df2f4be-0543-439c-8a7a-2235ad96dc42-config-volume\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:17.754089 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.754068    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf57ca8b-8820-478b-958a-b671cb621908-cert\") pod \"ingress-canary-tnknt\" (UID: \"bf57ca8b-8820-478b-958a-b671cb621908\") " pod="openshift-ingress-canary/ingress-canary-tnknt"
Sep 16 18:46:17.761424 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.761385    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz6nt\" (UniqueName: \"kubernetes.io/projected/7df2f4be-0543-439c-8a7a-2235ad96dc42-kube-api-access-nz6nt\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:17.761424 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.761421    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5dkq\" (UniqueName: \"kubernetes.io/projected/bf57ca8b-8820-478b-958a-b671cb621908-kube-api-access-b5dkq\") pod \"ingress-canary-tnknt\" (UID: \"bf57ca8b-8820-478b-958a-b671cb621908\") " pod="openshift-ingress-canary/ingress-canary-tnknt"
Sep 16 18:46:17.811617 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.811550    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tnknt"
Sep 16 18:46:17.991630 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.991594    2582 generic.go:356] "Generic (PLEG): container finished" podID="8ca622a1-571a-4769-a681-91f848125f1c" containerID="c111593f0e33e50a5c9010c13716f764f7fc4b523cb5d3b599948aab779c5ad9" exitCode=0
Sep 16 18:46:17.991630 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:17.991635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4twzx" event={"ID":"8ca622a1-571a-4769-a681-91f848125f1c","Type":"ContainerDied","Data":"c111593f0e33e50a5c9010c13716f764f7fc4b523cb5d3b599948aab779c5ad9"}
Sep 16 18:46:18.131659 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:18.131624    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tnknt"]
Sep 16 18:46:18.137930 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:46:18.137904    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf57ca8b_8820_478b_958a_b671cb621908.slice/crio-c16b2364f9efceb083e94e1c77fda077bcb233533171d04bfa722136beb6779c WatchSource:0}: Error finding container c16b2364f9efceb083e94e1c77fda077bcb233533171d04bfa722136beb6779c: Status 404 returned error can't find the container with id c16b2364f9efceb083e94e1c77fda077bcb233533171d04bfa722136beb6779c
Sep 16 18:46:18.254398 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:18.254295    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:18.254558 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:18.254418    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:18.254558 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:18.254473    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls podName:7df2f4be-0543-439c-8a7a-2235ad96dc42 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:19.254460274 +0000 UTC m=+37.085068797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls") pod "dns-default-tqdhb" (UID: "7df2f4be-0543-439c-8a7a-2235ad96dc42") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:18.702113 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:18.702083    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:18.703107 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:18.702183    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:18.703107 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:18.702289    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:18.705513 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:18.705485    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:46:18.705670 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:18.705521    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:46:18.707041 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:18.707015    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:46:18.707172 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:18.707061    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m77pk\""
Sep 16 18:46:18.707172 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:18.707075    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jpzh6\""
Sep 16 18:46:18.707172 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:18.707015    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:46:18.971927 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:18.971836    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:18.998934 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:18.998896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4twzx" event={"ID":"8ca622a1-571a-4769-a681-91f848125f1c","Type":"ContainerStarted","Data":"8284b66ba654a158b0702ac02c35e8683d085b72b777a9143c67721acbec208e"}
Sep 16 18:46:19.001036 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:19.001004    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tnknt" event={"ID":"bf57ca8b-8820-478b-958a-b671cb621908","Type":"ContainerStarted","Data":"c16b2364f9efceb083e94e1c77fda077bcb233533171d04bfa722136beb6779c"}
Sep 16 18:46:19.022307 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:19.022247    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4twzx" podStartSLOduration=7.201359857 podStartE2EDuration="36.022227899s" totalImagesPullingTime="28.820868042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:19.020749216 +0000 UTC m=+36.851357762" watchObservedRunningTime="2026-09-16 18:46:19.022227899 +0000 UTC m=+36.852836443"
Sep 16 18:46:19.073370 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:19.073331    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:19.172994 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:19.172923    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:19.263275 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:19.263171    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:19.263444 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:19.263356    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:19.263444 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:19.263440    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls podName:7df2f4be-0543-439c-8a7a-2235ad96dc42 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:21.263418316 +0000 UTC m=+39.094026853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls") pod "dns-default-tqdhb" (UID: "7df2f4be-0543-439c-8a7a-2235ad96dc42") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:19.272728 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:19.272688    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:21.008192 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.007957    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tnknt" event={"ID":"bf57ca8b-8820-478b-958a-b671cb621908","Type":"ContainerStarted","Data":"bc1ce5d07241e1a8fc62f80f86bf5132e9dbe132074e2232068dc121436b2763"}
Sep 16 18:46:21.024509 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.024457    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tnknt" podStartSLOduration=1.6544771360000001 podStartE2EDuration="4.024442214s" totalImagesPullingTime="2.369965078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:21.0234303 +0000 UTC m=+38.854038845" watchObservedRunningTime="2026-09-16 18:46:21.024442214 +0000 UTC m=+38.855050760"
Sep 16 18:46:21.279131 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.279099    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:21.279320 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:21.279251    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:21.279366 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:21.279324    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls podName:7df2f4be-0543-439c-8a7a-2235ad96dc42 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:25.27930938 +0000 UTC m=+43.109917903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls") pod "dns-default-tqdhb" (UID: "7df2f4be-0543-439c-8a7a-2235ad96dc42") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:21.313642 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.313607    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"]
Sep 16 18:46:21.329162 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.329124    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"]
Sep 16 18:46:21.329419 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.329393    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"
Sep 16 18:46:21.332470 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.332440    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5lgsr\""
Sep 16 18:46:21.332875 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.332851    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 18:46:21.332999 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.332858    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 18:46:21.480829 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.480797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf969786-dd12-4361-97db-549dec34aa9e-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-mt6zb\" (UID: \"bf969786-dd12-4361-97db-549dec34aa9e\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"
Sep 16 18:46:21.480829 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.480835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mt6zb\" (UID: \"bf969786-dd12-4361-97db-549dec34aa9e\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"
Sep 16 18:46:21.482437 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.482382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:21.542431 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.542314    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:21.581614 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.581551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf969786-dd12-4361-97db-549dec34aa9e-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-mt6zb\" (UID: \"bf969786-dd12-4361-97db-549dec34aa9e\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"
Sep 16 18:46:21.581614 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.581614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mt6zb\" (UID: \"bf969786-dd12-4361-97db-549dec34aa9e\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"
Sep 16 18:46:21.581837 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:21.581720    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:46:21.581837 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:21.581777    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert podName:bf969786-dd12-4361-97db-549dec34aa9e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:22.08176142 +0000 UTC m=+39.912369943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-mt6zb" (UID: "bf969786-dd12-4361-97db-549dec34aa9e") : secret "networking-console-plugin-cert" not found
Sep 16 18:46:21.582387 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:21.582359    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf969786-dd12-4361-97db-549dec34aa9e-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-mt6zb\" (UID: \"bf969786-dd12-4361-97db-549dec34aa9e\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"
Sep 16 18:46:22.085967 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:22.085927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mt6zb\" (UID: \"bf969786-dd12-4361-97db-549dec34aa9e\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"
Sep 16 18:46:22.086461 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:22.086062    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:46:22.086461 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:22.086122    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert podName:bf969786-dd12-4361-97db-549dec34aa9e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:23.086106807 +0000 UTC m=+40.916715330 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-mt6zb" (UID: "bf969786-dd12-4361-97db-549dec34aa9e") : secret "networking-console-plugin-cert" not found
Sep 16 18:46:22.627901 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:22.627851    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:22.685485 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:22.685439    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:22.929874 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:22.929793    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tbq7b"]
Sep 16 18:46:22.936561 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:22.936532    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:22.951777 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:22.951746    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tbq7b"]
Sep 16 18:46:22.951936 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:22.951877    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:22.954865 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:22.954839    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:46:22.955014 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:22.954839    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:46:22.955195 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:22.955178    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:46:22.956016 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:22.955998    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:46:22.956078 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:22.956002    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7cpmw\""
Sep 16 18:46:23.046654 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.046620    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.093845 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.093809    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.094192 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.093857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mt6zb\" (UID: \"bf969786-dd12-4361-97db-549dec34aa9e\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"
Sep 16 18:46:23.094192 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.093877    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0dff464f-883c-481c-8560-d49055bc580d-crio-socket\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.094192 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.093946    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dff464f-883c-481c-8560-d49055bc580d-data-volume\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.094192 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.093971    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.094192 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:23.093994    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:46:23.094192 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.094049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4pk8\" (UniqueName: \"kubernetes.io/projected/0dff464f-883c-481c-8560-d49055bc580d-kube-api-access-z4pk8\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.094192 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:23.094074    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert podName:bf969786-dd12-4361-97db-549dec34aa9e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:25.094055459 +0000 UTC m=+42.924663998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-mt6zb" (UID: "bf969786-dd12-4361-97db-549dec34aa9e") : secret "networking-console-plugin-cert" not found
Sep 16 18:46:23.147560 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.147504    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.194808 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.194722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.194808 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.194773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0dff464f-883c-481c-8560-d49055bc580d-crio-socket\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.195014 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:23.194843    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:23.195014 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.194929    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0dff464f-883c-481c-8560-d49055bc580d-crio-socket\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.195014 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:23.194936    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm podName:0dff464f-883c-481c-8560-d49055bc580d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:23.694917967 +0000 UTC m=+41.525526490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbq7b" (UID: "0dff464f-883c-481c-8560-d49055bc580d") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:23.195014 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.194918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dff464f-883c-481c-8560-d49055bc580d-data-volume\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.195014 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.194976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.195257 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.195088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4pk8\" (UniqueName: \"kubernetes.io/projected/0dff464f-883c-481c-8560-d49055bc580d-kube-api-access-z4pk8\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.195257 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:23.195129    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:23.195257 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:23.195194    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls podName:0dff464f-883c-481c-8560-d49055bc580d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:23.695178466 +0000 UTC m=+41.525786988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbq7b" (UID: "0dff464f-883c-481c-8560-d49055bc580d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:23.195257 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.195232    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dff464f-883c-481c-8560-d49055bc580d-data-volume\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.218464 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.218431    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4pk8\" (UniqueName: \"kubernetes.io/projected/0dff464f-883c-481c-8560-d49055bc580d-kube-api-access-z4pk8\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.249292 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.249259    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.550908 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.550874    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.652484 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.652437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.698897 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.698852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.699089 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.698929    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:23.699089 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:23.698987    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:23.699089 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:23.699035    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:23.699089 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:23.699066    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm podName:0dff464f-883c-481c-8560-d49055bc580d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:24.699043869 +0000 UTC m=+42.529652416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbq7b" (UID: "0dff464f-883c-481c-8560-d49055bc580d") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:23.699089 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:23.699089    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls podName:0dff464f-883c-481c-8560-d49055bc580d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:24.699073376 +0000 UTC m=+42.529681902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbq7b" (UID: "0dff464f-883c-481c-8560-d49055bc580d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:23.776612 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.776555    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.793236 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.793195    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-cdf2k"]
Sep 16 18:46:23.814534 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.814456    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-cdf2k"]
Sep 16 18:46:23.814695 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.814592    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cdf2k"
Sep 16 18:46:23.817813 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.817789    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2f4xv\""
Sep 16 18:46:23.817813 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.817808    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:46:23.818096 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.818080    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:23.889513 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.889477    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.952890 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:23.952859    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:24.001207 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:24.001164    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmxw4\" (UniqueName: \"kubernetes.io/projected/6357abcd-9ba6-4656-b180-15ac6a0cec77-kube-api-access-hmxw4\") pod \"migrator-8b4d697c5-cdf2k\" (UID: \"6357abcd-9ba6-4656-b180-15ac6a0cec77\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cdf2k"
Sep 16 18:46:24.101702 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:24.101561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmxw4\" (UniqueName: \"kubernetes.io/projected/6357abcd-9ba6-4656-b180-15ac6a0cec77-kube-api-access-hmxw4\") pod \"migrator-8b4d697c5-cdf2k\" (UID: \"6357abcd-9ba6-4656-b180-15ac6a0cec77\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cdf2k"
Sep 16 18:46:24.110929 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:24.110901    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmxw4\" (UniqueName: \"kubernetes.io/projected/6357abcd-9ba6-4656-b180-15ac6a0cec77-kube-api-access-hmxw4\") pod \"migrator-8b4d697c5-cdf2k\" (UID: \"6357abcd-9ba6-4656-b180-15ac6a0cec77\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cdf2k"
Sep 16 18:46:24.124023 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:24.123993    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cdf2k"
Sep 16 18:46:24.409566 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:24.409540    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-cdf2k"]
Sep 16 18:46:24.412143 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:46:24.412108    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6357abcd_9ba6_4656_b180_15ac6a0cec77.slice/crio-7f9a5f9bf242ab3902b6eb0c027ceb44014962daef423afe71527456b9e002b8 WatchSource:0}: Error finding container 7f9a5f9bf242ab3902b6eb0c027ceb44014962daef423afe71527456b9e002b8: Status 404 returned error can't find the container with id 7f9a5f9bf242ab3902b6eb0c027ceb44014962daef423afe71527456b9e002b8
Sep 16 18:46:24.705489 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:24.705409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:24.705489 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:24.705462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:24.705690 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:24.705537    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:24.705690 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:24.705620    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:24.705690 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:24.705627    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm podName:0dff464f-883c-481c-8560-d49055bc580d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:26.705609836 +0000 UTC m=+44.536218359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbq7b" (UID: "0dff464f-883c-481c-8560-d49055bc580d") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:24.705690 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:24.705660    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls podName:0dff464f-883c-481c-8560-d49055bc580d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:26.705649395 +0000 UTC m=+44.536257942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbq7b" (UID: "0dff464f-883c-481c-8560-d49055bc580d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:25.018790 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:25.018755    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cdf2k" event={"ID":"6357abcd-9ba6-4656-b180-15ac6a0cec77","Type":"ContainerStarted","Data":"7f9a5f9bf242ab3902b6eb0c027ceb44014962daef423afe71527456b9e002b8"}
Sep 16 18:46:25.067634 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:25.067603    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-stzn8_36f66f67-8c13-4880-9052-e0624f617592/dns-node-resolver/0.log"
Sep 16 18:46:25.109951 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:25.109907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mt6zb\" (UID: \"bf969786-dd12-4361-97db-549dec34aa9e\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"
Sep 16 18:46:25.110305 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:25.110065    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:46:25.110305 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:25.110132    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert podName:bf969786-dd12-4361-97db-549dec34aa9e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:29.11011458 +0000 UTC m=+46.940723103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-mt6zb" (UID: "bf969786-dd12-4361-97db-549dec34aa9e") : secret "networking-console-plugin-cert" not found
Sep 16 18:46:25.312039 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:25.311943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:25.312219 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:25.312132    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:25.312272 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:25.312225    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls podName:7df2f4be-0543-439c-8a7a-2235ad96dc42 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:33.312201433 +0000 UTC m=+51.142809958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls") pod "dns-default-tqdhb" (UID: "7df2f4be-0543-439c-8a7a-2235ad96dc42") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:25.464480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:25.464440    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:25.531144 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:25.531096    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:26.068694 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:26.068666    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z9b6q_9b5b5c2b-18f2-4855-bb08-5173767444bf/node-ca/0.log"
Sep 16 18:46:26.451283 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:26.451246    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:26.669217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:26.669140    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tnknt_bf57ca8b-8820-478b-958a-b671cb621908/serve-healthcheck-canary/0.log"
Sep 16 18:46:26.724788 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:26.724751    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:26.724996 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:26.724842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:26.724996 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:26.724908    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:26.724996 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:26.724919    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:26.724996 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:26.724963    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm podName:0dff464f-883c-481c-8560-d49055bc580d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:30.72494958 +0000 UTC m=+48.555558102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbq7b" (UID: "0dff464f-883c-481c-8560-d49055bc580d") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:26.724996 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:26.724978    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls podName:0dff464f-883c-481c-8560-d49055bc580d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:30.724971207 +0000 UTC m=+48.555579730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbq7b" (UID: "0dff464f-883c-481c-8560-d49055bc580d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:27.025361 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:27.025321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cdf2k" event={"ID":"6357abcd-9ba6-4656-b180-15ac6a0cec77","Type":"ContainerStarted","Data":"cffe69a5b34cd896884927956c80f68b383f95b1e9cc259a446604838fe3a7f6"}
Sep 16 18:46:27.025622 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:27.025381    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cdf2k" event={"ID":"6357abcd-9ba6-4656-b180-15ac6a0cec77","Type":"ContainerStarted","Data":"9214253c5025ff60df0828de29844e04793203dc722772c555af6087019deb18"}
Sep 16 18:46:27.043288 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:27.043232    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cdf2k" podStartSLOduration=2.344380371 podStartE2EDuration="4.043203252s" totalImagesPullingTime="1.698822881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:27.042392444 +0000 UTC m=+44.873000985" watchObservedRunningTime="2026-09-16 18:46:27.043203252 +0000 UTC m=+44.873811796"
Sep 16 18:46:27.338818 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:27.338714    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:27.475016 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:27.474970    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:27.537099 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:27.537062    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:27.594199 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:27.594090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:27.695165 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:27.695117    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:27.756880 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:27.756836    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:28.239305 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:28.239268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:28.438644 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:28.438608    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:29.145042 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:29.145003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mt6zb\" (UID: \"bf969786-dd12-4361-97db-549dec34aa9e\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"
Sep 16 18:46:29.145428 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:29.145158    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:46:29.145428 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:29.145227    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert podName:bf969786-dd12-4361-97db-549dec34aa9e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:37.145210752 +0000 UTC m=+54.975819275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-mt6zb" (UID: "bf969786-dd12-4361-97db-549dec34aa9e") : secret "networking-console-plugin-cert" not found
Sep 16 18:46:29.418066 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:29.417972    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:29.518252 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:29.518216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:29.642394 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:29.642355    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:30.043631 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:30.043593    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:30.240525 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:30.240487    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:30.757234 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:30.757186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:30.757419 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:30.757261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:30.757419 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:30.757308    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:30.757493 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:30.757417    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:30.757493 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:30.757425    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm podName:0dff464f-883c-481c-8560-d49055bc580d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:38.757408185 +0000 UTC m=+56.588016708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tbq7b" (UID: "0dff464f-883c-481c-8560-d49055bc580d") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:30.757493 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:30.757481    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls podName:0dff464f-883c-481c-8560-d49055bc580d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:38.757463742 +0000 UTC m=+56.588072278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tbq7b" (UID: "0dff464f-883c-481c-8560-d49055bc580d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:30.842973 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:30.842941    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.037697 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:31.037662    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.341961 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:31.341881    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.442414 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:31.442380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.635438 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:31.635355    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.858941 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:31.858911    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.917398 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:31.917315    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:32.236544 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:32.236467    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:32.789489 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:32.789458    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:32.890849 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:32.890817    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:33.240622 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:33.240516    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:33.376403 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:33.376367    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:33.378867 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:33.378846    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7df2f4be-0543-439c-8a7a-2235ad96dc42-metrics-tls\") pod \"dns-default-tqdhb\" (UID: \"7df2f4be-0543-439c-8a7a-2235ad96dc42\") " pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:33.387799 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:33.387760    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:33.544856 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:33.544823    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:33.646121 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:33.646081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:33.679885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:33.679812    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tqdhb"]
Sep 16 18:46:33.683360 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:46:33.683334    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df2f4be_0543_439c_8a7a_2235ad96dc42.slice/crio-8eb11892f0cd44018ce5a0f1edcaf2411f297ab62dc793c0c08c0c4615471d10 WatchSource:0}: Error finding container 8eb11892f0cd44018ce5a0f1edcaf2411f297ab62dc793c0c08c0c4615471d10: Status 404 returned error can't find the container with id 8eb11892f0cd44018ce5a0f1edcaf2411f297ab62dc793c0c08c0c4615471d10
Sep 16 18:46:33.835653 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:33.835540    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:34.044374 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:34.044344    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tqdhb" event={"ID":"7df2f4be-0543-439c-8a7a-2235ad96dc42","Type":"ContainerStarted","Data":"8eb11892f0cd44018ce5a0f1edcaf2411f297ab62dc793c0c08c0c4615471d10"}
Sep 16 18:46:34.437756 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:34.437715    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:34.993949 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:34.993917    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:35.275391 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:35.275363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:35.435828 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:35.435796    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:36.042816 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:36.042781    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:36.051586 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:36.051545    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tqdhb" event={"ID":"7df2f4be-0543-439c-8a7a-2235ad96dc42","Type":"ContainerStarted","Data":"6082628bb23bfbbde7f226dc365cee68d1c935c3bed9e740fa1727dbcf306a03"}
Sep 16 18:46:36.051749 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:36.051607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tqdhb" event={"ID":"7df2f4be-0543-439c-8a7a-2235ad96dc42","Type":"ContainerStarted","Data":"9a2caa86ac3f44d041c70cdf5a07aa0a7f9c1eac09ae6aeb9bf5713f8742fa50"}
Sep 16 18:46:36.051749 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:36.051640    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:36.069948 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:36.069900    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tqdhb" podStartSLOduration=17.564092922 podStartE2EDuration="19.069883579s" totalImagesPullingTime="1.505790657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:36.069754398 +0000 UTC m=+53.900362943" watchObservedRunningTime="2026-09-16 18:46:36.069883579 +0000 UTC m=+53.900492123"
Sep 16 18:46:36.146164 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:36.146135    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:36.445017 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:36.444931    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:36.839189 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:36.839157    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:37.208377 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:37.208290    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mt6zb\" (UID: \"bf969786-dd12-4361-97db-549dec34aa9e\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"
Sep 16 18:46:37.210901 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:37.210881    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf969786-dd12-4361-97db-549dec34aa9e-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-mt6zb\" (UID: \"bf969786-dd12-4361-97db-549dec34aa9e\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"
Sep 16 18:46:37.236454 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:37.236425    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:37.239445 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:37.239429    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"
Sep 16 18:46:37.436852 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:37.436814    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:37.504480 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:37.504423    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb"]
Sep 16 18:46:37.507516 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:46:37.507487    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf969786_dd12_4361_97db_549dec34aa9e.slice/crio-9d0d3a3d8b9298b5125a525a6476c4c5e679dbb8b8bc5bb2b182cd28eba1e584 WatchSource:0}: Error finding container 9d0d3a3d8b9298b5125a525a6476c4c5e679dbb8b8bc5bb2b182cd28eba1e584: Status 404 returned error can't find the container with id 9d0d3a3d8b9298b5125a525a6476c4c5e679dbb8b8bc5bb2b182cd28eba1e584
Sep 16 18:46:37.833218 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:37.833176    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:38.042095 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:38.042048    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:38.062786 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:38.062726    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb" event={"ID":"bf969786-dd12-4361-97db-549dec34aa9e","Type":"ContainerStarted","Data":"9d0d3a3d8b9298b5125a525a6476c4c5e679dbb8b8bc5bb2b182cd28eba1e584"}
Sep 16 18:46:38.825316 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:38.825206    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:38.825316 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:38.825261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:38.826074 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:38.826041    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0dff464f-883c-481c-8560-d49055bc580d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:38.827977 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:38.827957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0dff464f-883c-481c-8560-d49055bc580d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tbq7b\" (UID: \"0dff464f-883c-481c-8560-d49055bc580d\") " pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:38.860694 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:38.860636    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tbq7b"
Sep 16 18:46:39.068427 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:39.068386    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb" event={"ID":"bf969786-dd12-4361-97db-549dec34aa9e","Type":"ContainerStarted","Data":"4ec31c55ea8ec5049de2d4fef10fdc3f7796470d4169c1e9c7985f00f0d6eec6"}
Sep 16 18:46:39.103591 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:39.103434    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-mt6zb" podStartSLOduration=17.036362109 podStartE2EDuration="18.103410433s" totalImagesPullingTime="1.067048324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:39.102919023 +0000 UTC m=+56.933527559" watchObservedRunningTime="2026-09-16 18:46:39.103410433 +0000 UTC m=+56.934018979"
Sep 16 18:46:39.136491 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:39.136454    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tbq7b"]
Sep 16 18:46:39.141066 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:46:39.141031    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dff464f_883c_481c_8560_d49055bc580d.slice/crio-65d6a580a73b34886d65e36625afdbab280ba6ececd3e4827af340af5e3669c7 WatchSource:0}: Error finding container 65d6a580a73b34886d65e36625afdbab280ba6ececd3e4827af340af5e3669c7: Status 404 returned error can't find the container with id 65d6a580a73b34886d65e36625afdbab280ba6ececd3e4827af340af5e3669c7
Sep 16 18:46:40.072595 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:40.072536    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbq7b" event={"ID":"0dff464f-883c-481c-8560-d49055bc580d","Type":"ContainerStarted","Data":"85912991272bcd89f849d30fd9748ed10a5f18c37a3a9e23083f53daba935b83"}
Sep 16 18:46:40.072990 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:40.072616    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbq7b" event={"ID":"0dff464f-883c-481c-8560-d49055bc580d","Type":"ContainerStarted","Data":"65d6a580a73b34886d65e36625afdbab280ba6ececd3e4827af340af5e3669c7"}
Sep 16 18:46:41.083396 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:41.083347    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbq7b" event={"ID":"0dff464f-883c-481c-8560-d49055bc580d","Type":"ContainerStarted","Data":"c4a1b187ad2328bbde89064d506d9ce3090de32eae74aa65ca53da5fec125eae"}
Sep 16 18:46:42.089086 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:42.089040    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tbq7b" event={"ID":"0dff464f-883c-481c-8560-d49055bc580d","Type":"ContainerStarted","Data":"f1d5ceee406cae4e9c6527d25f115a383564d8f52ef287a88c76140f45340d23"}
Sep 16 18:46:42.107938 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:42.107884    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tbq7b" podStartSLOduration=17.705802856 podStartE2EDuration="20.107867135s" totalImagesPullingTime="2.402064279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:42.107317291 +0000 UTC m=+59.937925814" watchObservedRunningTime="2026-09-16 18:46:42.107867135 +0000 UTC m=+59.938475682"
Sep 16 18:46:42.985634 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:42.985596    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dvn2p"
Sep 16 18:46:46.059638 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:46.059608    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tqdhb"
Sep 16 18:46:46.982246 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:46.982212    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:47.047974 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:47.047937    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:48.502097 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.502050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs\") pod \"network-metrics-daemon-2wlpl\" (UID: \"ae831f85-3302-4b52-8cea-c9f437a9409e\") " pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:48.502539 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.502110    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:48.505017 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.504993    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:46:48.505156 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.504993    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:46:48.514946 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.514908    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f0c80db-ca40-4eec-b140-a8e43262ed47-original-pull-secret\") pod \"global-pull-secret-syncer-bdq8w\" (UID: \"5f0c80db-ca40-4eec-b140-a8e43262ed47\") " pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:48.515072 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.514906    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae831f85-3302-4b52-8cea-c9f437a9409e-metrics-certs\") pod \"network-metrics-daemon-2wlpl\" (UID: \"ae831f85-3302-4b52-8cea-c9f437a9409e\") " pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:48.602456 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.602413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx2bd\" (UniqueName: \"kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd\") pod \"network-check-target-jsfns\" (UID: \"8339946f-2f59-4fce-93a4-a2a320b11588\") " pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:48.605391 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.605372    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:46:48.615509 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.615479    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:46:48.626386 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.626358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx2bd\" (UniqueName: \"kubernetes.io/projected/8339946f-2f59-4fce-93a4-a2a320b11588-kube-api-access-mx2bd\") pod \"network-check-target-jsfns\" (UID: \"8339946f-2f59-4fce-93a4-a2a320b11588\") " pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:48.718985 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.718953    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m77pk\""
Sep 16 18:46:48.722160 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.722142    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdq8w"
Sep 16 18:46:48.727017 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.726986    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wlpl"
Sep 16 18:46:48.733372 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.733347    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jpzh6\""
Sep 16 18:46:48.739901 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.739871    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:48.999499 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:48.999465    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bdq8w"]
Sep 16 18:46:49.002559 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:46:49.002532    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0c80db_ca40_4eec_b140_a8e43262ed47.slice/crio-3c626e3aa5b569bf0bcbfc59de4ebfb634126249eef8090048087def6ce95f50 WatchSource:0}: Error finding container 3c626e3aa5b569bf0bcbfc59de4ebfb634126249eef8090048087def6ce95f50: Status 404 returned error can't find the container with id 3c626e3aa5b569bf0bcbfc59de4ebfb634126249eef8090048087def6ce95f50
Sep 16 18:46:49.010959 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:49.010931    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2wlpl"]
Sep 16 18:46:49.042514 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:49.042439    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jsfns"]
Sep 16 18:46:49.046384 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:46:49.046355    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8339946f_2f59_4fce_93a4_a2a320b11588.slice/crio-7e315c2bd0f8ee2002342d31ded60173853894634b0972d4dd0d7ae8e891919a WatchSource:0}: Error finding container 7e315c2bd0f8ee2002342d31ded60173853894634b0972d4dd0d7ae8e891919a: Status 404 returned error can't find the container with id 7e315c2bd0f8ee2002342d31ded60173853894634b0972d4dd0d7ae8e891919a
Sep 16 18:46:49.111717 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:49.111672    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2wlpl" event={"ID":"ae831f85-3302-4b52-8cea-c9f437a9409e","Type":"ContainerStarted","Data":"305a642cd0120b7de93d5ddc389f37d7c96f3cb5481dc7ee78c35807c176cf94"}
Sep 16 18:46:49.112994 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:49.112968    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jsfns" event={"ID":"8339946f-2f59-4fce-93a4-a2a320b11588","Type":"ContainerStarted","Data":"7e315c2bd0f8ee2002342d31ded60173853894634b0972d4dd0d7ae8e891919a"}
Sep 16 18:46:49.114120 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:49.114098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bdq8w" event={"ID":"5f0c80db-ca40-4eec-b140-a8e43262ed47","Type":"ContainerStarted","Data":"3c626e3aa5b569bf0bcbfc59de4ebfb634126249eef8090048087def6ce95f50"}
Sep 16 18:46:50.038832 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.038551    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"]
Sep 16 18:46:50.043759 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.043732    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"
Sep 16 18:46:50.049217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.048782    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769bfc7686-lfkc8"]
Sep 16 18:46:50.049217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.049137    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 18:46:50.049436 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.049407    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:46:50.049436 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.049063    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:46:50.049619 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.049050    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:46:50.051935 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.051913    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769bfc7686-lfkc8"
Sep 16 18:46:50.053448 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.053427    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"]
Sep 16 18:46:50.055442 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.055420    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-dt8zd\""
Sep 16 18:46:50.055742 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.055724    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:46:50.064156 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.064127    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769bfc7686-lfkc8"]
Sep 16 18:46:50.113272 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.112983    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b40c03cf-9bb3-49ad-bd3f-d9c6f2267c3f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-769bfc7686-lfkc8\" (UID: \"b40c03cf-9bb3-49ad-bd3f-d9c6f2267c3f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769bfc7686-lfkc8"
Sep 16 18:46:50.113272 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.113028    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crm56\" (UniqueName: \"kubernetes.io/projected/b40c03cf-9bb3-49ad-bd3f-d9c6f2267c3f-kube-api-access-crm56\") pod \"managed-serviceaccount-addon-agent-769bfc7686-lfkc8\" (UID: \"b40c03cf-9bb3-49ad-bd3f-d9c6f2267c3f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769bfc7686-lfkc8"
Sep 16 18:46:50.113272 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.113057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b9db5dc-2530-4f33-a772-c3a9c0eafe51-tmp\") pod \"klusterlet-addon-workmgr-6466fc4d69-vmbrg\" (UID: \"9b9db5dc-2530-4f33-a772-c3a9c0eafe51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"
Sep 16 18:46:50.113272 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.113099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9b9db5dc-2530-4f33-a772-c3a9c0eafe51-klusterlet-config\") pod \"klusterlet-addon-workmgr-6466fc4d69-vmbrg\" (UID: \"9b9db5dc-2530-4f33-a772-c3a9c0eafe51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"
Sep 16 18:46:50.113272 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.113129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbq29\" (UniqueName: \"kubernetes.io/projected/9b9db5dc-2530-4f33-a772-c3a9c0eafe51-kube-api-access-hbq29\") pod \"klusterlet-addon-workmgr-6466fc4d69-vmbrg\" (UID: \"9b9db5dc-2530-4f33-a772-c3a9c0eafe51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"
Sep 16 18:46:50.139201 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.138006    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-99b67cb9-kcqjd"]
Sep 16 18:46:50.142454 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.141547    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.142454 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.142160    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:50.146772 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.146666    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:46:50.147469 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.147320    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bk56d\""
Sep 16 18:46:50.147469 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.147340    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:46:50.148212 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.147773    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:46:50.153357 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.153166    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:46:50.158693 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.158640    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-99b67cb9-kcqjd"]
Sep 16 18:46:50.214220 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.214185    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-installation-pull-secrets\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.214354 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.214245    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnrdd\" (UniqueName: \"kubernetes.io/projected/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-kube-api-access-dnrdd\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.214354 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.214279    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b40c03cf-9bb3-49ad-bd3f-d9c6f2267c3f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-769bfc7686-lfkc8\" (UID: \"b40c03cf-9bb3-49ad-bd3f-d9c6f2267c3f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769bfc7686-lfkc8"
Sep 16 18:46:50.214354 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.214297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crm56\" (UniqueName: \"kubernetes.io/projected/b40c03cf-9bb3-49ad-bd3f-d9c6f2267c3f-kube-api-access-crm56\") pod \"managed-serviceaccount-addon-agent-769bfc7686-lfkc8\" (UID: \"b40c03cf-9bb3-49ad-bd3f-d9c6f2267c3f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769bfc7686-lfkc8"
Sep 16 18:46:50.214354 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.214316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b9db5dc-2530-4f33-a772-c3a9c0eafe51-tmp\") pod \"klusterlet-addon-workmgr-6466fc4d69-vmbrg\" (UID: \"9b9db5dc-2530-4f33-a772-c3a9c0eafe51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"
Sep 16 18:46:50.214354 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.214337    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-registry-certificates\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.214647 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.214362    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9b9db5dc-2530-4f33-a772-c3a9c0eafe51-klusterlet-config\") pod \"klusterlet-addon-workmgr-6466fc4d69-vmbrg\" (UID: \"9b9db5dc-2530-4f33-a772-c3a9c0eafe51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"
Sep 16 18:46:50.214647 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.214378    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-bound-sa-token\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.214647 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.214397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbq29\" (UniqueName: \"kubernetes.io/projected/9b9db5dc-2530-4f33-a772-c3a9c0eafe51-kube-api-access-hbq29\") pod \"klusterlet-addon-workmgr-6466fc4d69-vmbrg\" (UID: \"9b9db5dc-2530-4f33-a772-c3a9c0eafe51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"
Sep 16 18:46:50.214647 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.214412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-image-registry-private-configuration\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.214647 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.214447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-registry-tls\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.214647 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.214476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-ca-trust-extracted\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.214647 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.214499    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-trusted-ca\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.215717 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.215257    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b9db5dc-2530-4f33-a772-c3a9c0eafe51-tmp\") pod \"klusterlet-addon-workmgr-6466fc4d69-vmbrg\" (UID: \"9b9db5dc-2530-4f33-a772-c3a9c0eafe51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"
Sep 16 18:46:50.218041 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.218011    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b40c03cf-9bb3-49ad-bd3f-d9c6f2267c3f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-769bfc7686-lfkc8\" (UID: \"b40c03cf-9bb3-49ad-bd3f-d9c6f2267c3f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769bfc7686-lfkc8"
Sep 16 18:46:50.219761 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.219713    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9b9db5dc-2530-4f33-a772-c3a9c0eafe51-klusterlet-config\") pod \"klusterlet-addon-workmgr-6466fc4d69-vmbrg\" (UID: \"9b9db5dc-2530-4f33-a772-c3a9c0eafe51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"
Sep 16 18:46:50.225038 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.224497    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbq29\" (UniqueName: \"kubernetes.io/projected/9b9db5dc-2530-4f33-a772-c3a9c0eafe51-kube-api-access-hbq29\") pod \"klusterlet-addon-workmgr-6466fc4d69-vmbrg\" (UID: \"9b9db5dc-2530-4f33-a772-c3a9c0eafe51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"
Sep 16 18:46:50.225038 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.224988    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crm56\" (UniqueName: \"kubernetes.io/projected/b40c03cf-9bb3-49ad-bd3f-d9c6f2267c3f-kube-api-access-crm56\") pod \"managed-serviceaccount-addon-agent-769bfc7686-lfkc8\" (UID: \"b40c03cf-9bb3-49ad-bd3f-d9c6f2267c3f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769bfc7686-lfkc8"
Sep 16 18:46:50.239753 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.239614    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:50.315041 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.314974    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnrdd\" (UniqueName: \"kubernetes.io/projected/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-kube-api-access-dnrdd\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.315041 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.315023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-registry-certificates\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.315347 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.315069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-bound-sa-token\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.315347 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.315097    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-image-registry-private-configuration\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.315347 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.315140    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-registry-tls\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.315347 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.315166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-ca-trust-extracted\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.315347 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.315188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-trusted-ca\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.315347 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.315233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-installation-pull-secrets\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.316652 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.316217    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-registry-certificates\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.316816 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.316721    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-ca-trust-extracted\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.316884 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.316849    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-trusted-ca\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.319540 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.319495    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-image-registry-private-configuration\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.320348 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.320002    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-installation-pull-secrets\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.320541 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.320520    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-registry-tls\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.325413 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.325384    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnrdd\" (UniqueName: \"kubernetes.io/projected/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-kube-api-access-dnrdd\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.328041 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.327835    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f41b655b-8c93-4675-ae6b-6ac4cfebbf58-bound-sa-token\") pod \"image-registry-99b67cb9-kcqjd\" (UID: \"f41b655b-8c93-4675-ae6b-6ac4cfebbf58\") " pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.338202 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.338170    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:50.357702 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.357470    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"
Sep 16 18:46:50.373527 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.373491    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769bfc7686-lfkc8"
Sep 16 18:46:50.438245 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.438174    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:50.467817 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.467781    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:50.653806 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.653712    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"]
Sep 16 18:46:50.658515 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:46:50.658475    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9db5dc_2530_4f33_a772_c3a9c0eafe51.slice/crio-93a318a72a7dcd311324e27bb3195b8feaf3b6f40dd0d2bbe73ba864324fb59d WatchSource:0}: Error finding container 93a318a72a7dcd311324e27bb3195b8feaf3b6f40dd0d2bbe73ba864324fb59d: Status 404 returned error can't find the container with id 93a318a72a7dcd311324e27bb3195b8feaf3b6f40dd0d2bbe73ba864324fb59d
Sep 16 18:46:50.678961 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.678921    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769bfc7686-lfkc8"]
Sep 16 18:46:50.762090 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:50.762041    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-99b67cb9-kcqjd"]
Sep 16 18:46:50.766320 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:46:50.766278    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41b655b_8c93_4675_ae6b_6ac4cfebbf58.slice/crio-7a5e60d851ea9b0745a3ae068e1c757db1f263ffbad186c83348265eaebac8df WatchSource:0}: Error finding container 7a5e60d851ea9b0745a3ae068e1c757db1f263ffbad186c83348265eaebac8df: Status 404 returned error can't find the container with id 7a5e60d851ea9b0745a3ae068e1c757db1f263ffbad186c83348265eaebac8df
Sep 16 18:46:51.117193 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:51.117153    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.127026 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:51.126894    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2wlpl" event={"ID":"ae831f85-3302-4b52-8cea-c9f437a9409e","Type":"ContainerStarted","Data":"7c6a0f9adf1e57cbce15a93a8f898e197511689c42de5dacee959fd5a85fb3f1"}
Sep 16 18:46:51.127026 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:51.126955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2wlpl" event={"ID":"ae831f85-3302-4b52-8cea-c9f437a9409e","Type":"ContainerStarted","Data":"65b844500419b1587e89279e45d1722922c19d4b90dec51d536d7755b4191084"}
Sep 16 18:46:51.128823 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:51.128789    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769bfc7686-lfkc8" event={"ID":"b40c03cf-9bb3-49ad-bd3f-d9c6f2267c3f","Type":"ContainerStarted","Data":"6bb31f742784b699e14161d1cd33959ca105b661e099e6f6f584ede9ddb53a6f"}
Sep 16 18:46:51.131002 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:51.130968    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-99b67cb9-kcqjd" event={"ID":"f41b655b-8c93-4675-ae6b-6ac4cfebbf58","Type":"ContainerStarted","Data":"46c0c14ec5c593f4caebc257fb7460301ac4ae2e73171d7c1590dd305e5d8cd6"}
Sep 16 18:46:51.131139 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:51.131023    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-99b67cb9-kcqjd" event={"ID":"f41b655b-8c93-4675-ae6b-6ac4cfebbf58","Type":"ContainerStarted","Data":"7a5e60d851ea9b0745a3ae068e1c757db1f263ffbad186c83348265eaebac8df"}
Sep 16 18:46:51.131615 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:51.131566    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:46:51.133586 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:51.133553    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg" event={"ID":"9b9db5dc-2530-4f33-a772-c3a9c0eafe51","Type":"ContainerStarted","Data":"93a318a72a7dcd311324e27bb3195b8feaf3b6f40dd0d2bbe73ba864324fb59d"}
Sep 16 18:46:51.145957 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:51.145907    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2wlpl" podStartSLOduration=67.084099887 podStartE2EDuration="1m8.145887121s" totalImagesPullingTime="1.061787234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:51.143673345 +0000 UTC m=+68.974281890" watchObservedRunningTime="2026-09-16 18:46:51.145887121 +0000 UTC m=+68.976495667"
Sep 16 18:46:51.257900 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:51.257844    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.327972 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:51.327932    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.583315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:51.583243    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.684116 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:51.683959    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.740626 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:51.740505    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:54.293439 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:54.293403    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:54.394503 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:54.394468    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:54.494822 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:54.494772    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:56.390976 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:56.390940    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:58.146474 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.146438    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:58.164972 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.164918    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg" event={"ID":"9b9db5dc-2530-4f33-a772-c3a9c0eafe51","Type":"ContainerStarted","Data":"abab6a3a3144e5dc0554781e797106dc04039407a7672dbe9be4b8b1876d6b25"}
Sep 16 18:46:58.165179 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.165124    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"
Sep 16 18:46:58.167093 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.167069    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg"
Sep 16 18:46:58.167245 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.167114    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769bfc7686-lfkc8" event={"ID":"b40c03cf-9bb3-49ad-bd3f-d9c6f2267c3f","Type":"ContainerStarted","Data":"eefa2471ce093d02fe3a484cc679d3b1f1d3893f94617fb989b0a20141b908b9"}
Sep 16 18:46:58.169232 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.169206    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jsfns" event={"ID":"8339946f-2f59-4fce-93a4-a2a320b11588","Type":"ContainerStarted","Data":"9277b2fd111232f2df6eb4ab77588f6dda72ad47567ee02fa21c80a944e024e2"}
Sep 16 18:46:58.169367 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.169283    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:46:58.170877 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.170854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bdq8w" event={"ID":"5f0c80db-ca40-4eec-b140-a8e43262ed47","Type":"ContainerStarted","Data":"483a68f65c22146499b8e0baefed9dd47bfe8fcc5c6f8026cf991e9ab9ee9c98"}
Sep 16 18:46:58.182499 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.182450    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6466fc4d69-vmbrg" podStartSLOduration=1.2775446750000001 podStartE2EDuration="8.182433641s" totalImagesPullingTime="6.904888966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:58.181373816 +0000 UTC m=+76.011982361" watchObservedRunningTime="2026-09-16 18:46:58.182433641 +0000 UTC m=+76.013042185"
Sep 16 18:46:58.182705 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.182685    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-99b67cb9-kcqjd" podStartSLOduration=8.182678215 podStartE2EDuration="8.182678215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:51.162489324 +0000 UTC m=+68.993097872" watchObservedRunningTime="2026-09-16 18:46:58.182678215 +0000 UTC m=+76.013286757"
Sep 16 18:46:58.203396 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.203360    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:58.213456 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.213408    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769bfc7686-lfkc8" podStartSLOduration=1.326012173 podStartE2EDuration="8.213391436s" totalImagesPullingTime="6.887379263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:58.212528498 +0000 UTC m=+76.043137055" watchObservedRunningTime="2026-09-16 18:46:58.213391436 +0000 UTC m=+76.043999984"
Sep 16 18:46:58.231101 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.231046    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bdq8w" podStartSLOduration=66.723642249 podStartE2EDuration="1m15.231028653s" totalImagesPullingTime="8.507386404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:58.230508727 +0000 UTC m=+76.061117273" watchObservedRunningTime="2026-09-16 18:46:58.231028653 +0000 UTC m=+76.061637197"
Sep 16 18:46:58.247525 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.247467    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jsfns" podStartSLOduration=66.78471637 podStartE2EDuration="1m15.247450919s" totalImagesPullingTime="8.462734549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:58.246814429 +0000 UTC m=+76.077422974" watchObservedRunningTime="2026-09-16 18:46:58.247450919 +0000 UTC m=+76.078059464"
Sep 16 18:46:58.429506 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.429408    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:58.646667 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.646628    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:58.711815 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.711729    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pmg92"]
Sep 16 18:46:58.714977 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.714961    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.718071 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.718043    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:46:58.718071 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.718056    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:46:58.718281 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.718099    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:46:58.719349 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.719328    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:46:58.719455 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.719358    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kdm59\""
Sep 16 18:46:58.719455 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.719330    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:46:58.719455 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.719329    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:46:58.748653 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.748614    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:58.781762 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.781726    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f4150c27-bace-4fce-b004-8fb46d002878-root\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.781934 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.781768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-accelerators-collector-config\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.781934 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.781827    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-tls\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.781934 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.781859    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65mwz\" (UniqueName: \"kubernetes.io/projected/f4150c27-bace-4fce-b004-8fb46d002878-kube-api-access-65mwz\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.781934 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.781925    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-wtmp\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.782090 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.781952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.782090 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.781981    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-textfile\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.782090 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.782016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4150c27-bace-4fce-b004-8fb46d002878-metrics-client-ca\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.782090 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.782040    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4150c27-bace-4fce-b004-8fb46d002878-sys\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.807502 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.807466    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:58.882424 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.882383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-textfile\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.882424 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.882422    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4150c27-bace-4fce-b004-8fb46d002878-metrics-client-ca\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.882742 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.882444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4150c27-bace-4fce-b004-8fb46d002878-sys\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.882742 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.882469    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f4150c27-bace-4fce-b004-8fb46d002878-root\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.882742 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.882500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-accelerators-collector-config\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.882742 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.882535    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4150c27-bace-4fce-b004-8fb46d002878-sys\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.882742 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.882551    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f4150c27-bace-4fce-b004-8fb46d002878-root\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.882742 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.882540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-tls\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.882742 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.882627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65mwz\" (UniqueName: \"kubernetes.io/projected/f4150c27-bace-4fce-b004-8fb46d002878-kube-api-access-65mwz\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.882742 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.882661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-wtmp\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.882742 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.882687    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.883223 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.882852    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-textfile\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.883223 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.882867    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-wtmp\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.883223 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.883164    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-accelerators-collector-config\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.883223 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.883195    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4150c27-bace-4fce-b004-8fb46d002878-metrics-client-ca\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.885346 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.885318    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-tls\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.885449 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.885426    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4150c27-bace-4fce-b004-8fb46d002878-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.904094 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.904059    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65mwz\" (UniqueName: \"kubernetes.io/projected/f4150c27-bace-4fce-b004-8fb46d002878-kube-api-access-65mwz\") pod \"node-exporter-pmg92\" (UID: \"f4150c27-bace-4fce-b004-8fb46d002878\") " pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:58.931949 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:58.931910    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.018198 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.018171    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.025126 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.025100    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pmg92"
Sep 16 18:46:59.037152 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:46:59.037126    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4150c27_bace_4fce_b004_8fb46d002878.slice/crio-2fa7cf50accd777b83b2095566659bc4876dc4ef0ba112998410f938b41901a5 WatchSource:0}: Error finding container 2fa7cf50accd777b83b2095566659bc4876dc4ef0ba112998410f938b41901a5: Status 404 returned error can't find the container with id 2fa7cf50accd777b83b2095566659bc4876dc4ef0ba112998410f938b41901a5
Sep 16 18:46:59.122551 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.122514    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.175808 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.175771    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmg92" event={"ID":"f4150c27-bace-4fce-b004-8fb46d002878","Type":"ContainerStarted","Data":"2fa7cf50accd777b83b2095566659bc4876dc4ef0ba112998410f938b41901a5"}
Sep 16 18:46:59.178814 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.178781    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.485444 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.485350    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.653148 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.653112    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.720516 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.720467    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.724899 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.724859    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:59.728944 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.728922    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.732248 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.732218    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:46:59.732391 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.732272    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:46:59.732391 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.732294    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ddwbj\""
Sep 16 18:46:59.732524 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.732511    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:46:59.732637 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.732617    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:46:59.732785 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.732756    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:46:59.733565 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.733551    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:46:59.733846 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.733829    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:46:59.733953 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.733834    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:46:59.733953 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.733835    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:46:59.745980 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.745953    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:59.788885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.788852    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.788885 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.788889    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.789157 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.788926    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-web-config\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.789157 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.789006    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.789157 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.789047    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-config-volume\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.789157 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.789081    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.789157 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.789103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5x27\" (UniqueName: \"kubernetes.io/projected/c448a1cc-5061-4290-99d4-39c846818630-kube-api-access-x5x27\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.789157 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.789133    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.789157 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.789158    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c448a1cc-5061-4290-99d4-39c846818630-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.789518 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.789230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.789518 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.789256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.789518 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.789284    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c448a1cc-5061-4290-99d4-39c846818630-config-out\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.789518 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.789312    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.865928 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.865859    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.890008 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.889962    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.890220 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.890020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.890220 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.890051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c448a1cc-5061-4290-99d4-39c846818630-config-out\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.890220 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.890089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.890220 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.890117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.890220 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.890147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.890220 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.890185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-web-config\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.890220 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.890213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.890652 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.890245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-config-volume\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.890652 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.890286    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.890652 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.890309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5x27\" (UniqueName: \"kubernetes.io/projected/c448a1cc-5061-4290-99d4-39c846818630-kube-api-access-x5x27\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.890652 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.890337    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.890652 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.890370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c448a1cc-5061-4290-99d4-39c846818630-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.891055 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:46:59.890658    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-trusted-ca-bundle podName:c448a1cc-5061-4290-99d4-39c846818630 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:00.390631053 +0000 UTC m=+78.221239579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "c448a1cc-5061-4290-99d4-39c846818630") : configmap references non-existent config key: ca-bundle.crt
Sep 16 18:46:59.891567 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.891534    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.892118 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.892084    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.893890 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.893833    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.893890 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.893872    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c448a1cc-5061-4290-99d4-39c846818630-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.894352 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.894307    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.894566 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.894545    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.895027 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.894977    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c448a1cc-5061-4290-99d4-39c846818630-config-out\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.895142 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.895052    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-web-config\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.895441 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.895419    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.896399 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.896377    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-config-volume\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.896502 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.896436    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:59.900078 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:46:59.900058    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5x27\" (UniqueName: \"kubernetes.io/projected/c448a1cc-5061-4290-99d4-39c846818630-kube-api-access-x5x27\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:00.181290 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:00.181250    2582 generic.go:356] "Generic (PLEG): container finished" podID="f4150c27-bace-4fce-b004-8fb46d002878" containerID="26662fe1b3f45f00b59440caae77614338ed6e2350e53a9b38dcf2b4acebaa2e" exitCode=0
Sep 16 18:47:00.181736 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:00.181339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmg92" event={"ID":"f4150c27-bace-4fce-b004-8fb46d002878","Type":"ContainerDied","Data":"26662fe1b3f45f00b59440caae77614338ed6e2350e53a9b38dcf2b4acebaa2e"}
Sep 16 18:47:00.395107 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:00.395064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:00.396054 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:00.396032    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:00.639494 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:00.639463    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:00.922030 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:00.921999    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:00.923259 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:47:00.923233    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc448a1cc_5061_4290_99d4_39c846818630.slice/crio-b26185e1d284f40ac56a1badcc17e36403237904f18ea8c9f2712fc5fd0a253a WatchSource:0}: Error finding container b26185e1d284f40ac56a1badcc17e36403237904f18ea8c9f2712fc5fd0a253a: Status 404 returned error can't find the container with id b26185e1d284f40ac56a1badcc17e36403237904f18ea8c9f2712fc5fd0a253a
Sep 16 18:47:01.189715 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.189591    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmg92" event={"ID":"f4150c27-bace-4fce-b004-8fb46d002878","Type":"ContainerStarted","Data":"5c70e272b5846c4c18edf8080528e14b261621a67da86104d7ed817ca73ebb63"}
Sep 16 18:47:01.189715 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.189675    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmg92" event={"ID":"f4150c27-bace-4fce-b004-8fb46d002878","Type":"ContainerStarted","Data":"86bf293f0adc0cab999de5866da8d0e8748d75320017164ccc45f42dcd553c7b"}
Sep 16 18:47:01.190898 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.190869    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerStarted","Data":"b26185e1d284f40ac56a1badcc17e36403237904f18ea8c9f2712fc5fd0a253a"}
Sep 16 18:47:01.208784 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.208255    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pmg92" podStartSLOduration=2.190961329 podStartE2EDuration="3.20822514s" totalImagesPullingTime="945.190457ms" totalInitContainerRuntime="72.073354ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:01.207531084 +0000 UTC m=+79.038139663" watchObservedRunningTime="2026-09-16 18:47:01.20822514 +0000 UTC m=+79.038833685"
Sep 16 18:47:01.620135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.620094    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:01.685845 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.685809    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6fd4d644d6-42989"]
Sep 16 18:47:01.690139 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.690106    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.693547 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.693282    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 18:47:01.693547 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.693289    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 18:47:01.693547 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.693390    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 18:47:01.693547 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.693391    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b0h9vcm9qevm6\""
Sep 16 18:47:01.693547 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.693459    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 18:47:01.694072 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.694053    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 18:47:01.694336 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.694316    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6tl6x\""
Sep 16 18:47:01.702005 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.701968    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6fd4d644d6-42989"]
Sep 16 18:47:01.710299 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.710263    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:01.805341 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.805306    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.805505 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.805369    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.805505 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.805424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.805505 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.805461    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.805505 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.805490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hggr\" (UniqueName: \"kubernetes.io/projected/306b52a5-f1ec-4522-bd1d-c5fb0430f230-kube-api-access-8hggr\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.805689 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.805546    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-tls\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.805689 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.805638    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-grpc-tls\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.805689 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.805665    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/306b52a5-f1ec-4522-bd1d-c5fb0430f230-metrics-client-ca\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.906672 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.906533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hggr\" (UniqueName: \"kubernetes.io/projected/306b52a5-f1ec-4522-bd1d-c5fb0430f230-kube-api-access-8hggr\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.906672 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.906616    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-tls\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.906672 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.906661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-grpc-tls\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.906998 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.906692    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/306b52a5-f1ec-4522-bd1d-c5fb0430f230-metrics-client-ca\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.906998 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.906726    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.906998 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.906778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.906998 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.906826    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.906998 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.906864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.907981 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.907942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/306b52a5-f1ec-4522-bd1d-c5fb0430f230-metrics-client-ca\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.911633 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.911478    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-tls\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.911801 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.911124    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.911875 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.911822    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.915606 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.912240    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.919007 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.918970    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.919736 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.919677    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/306b52a5-f1ec-4522-bd1d-c5fb0430f230-secret-grpc-tls\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:01.921075 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:01.921051    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hggr\" (UniqueName: \"kubernetes.io/projected/306b52a5-f1ec-4522-bd1d-c5fb0430f230-kube-api-access-8hggr\") pod \"thanos-querier-6fd4d644d6-42989\" (UID: \"306b52a5-f1ec-4522-bd1d-c5fb0430f230\") " pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:02.004910 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:02.004859    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:02.087128 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:02.086932    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:02.197450 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:02.197341    2582 generic.go:356] "Generic (PLEG): container finished" podID="c448a1cc-5061-4290-99d4-39c846818630" containerID="619f57411785059460784b20c17188adea750b98387894e2c258c06ebbe6061a" exitCode=0
Sep 16 18:47:02.197450 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:02.197426    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerDied","Data":"619f57411785059460784b20c17188adea750b98387894e2c258c06ebbe6061a"}
Sep 16 18:47:02.208239 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:02.208206    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:02.294983 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:02.294937    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:02.319276 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:02.319246    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6fd4d644d6-42989"]
Sep 16 18:47:02.322233 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:47:02.322203    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306b52a5_f1ec_4522_bd1d_c5fb0430f230.slice/crio-09698f7b8facc0a168245473925a6b82f56c499c38941f213a4f7a5d87639c10 WatchSource:0}: Error finding container 09698f7b8facc0a168245473925a6b82f56c499c38941f213a4f7a5d87639c10: Status 404 returned error can't find the container with id 09698f7b8facc0a168245473925a6b82f56c499c38941f213a4f7a5d87639c10
Sep 16 18:47:02.362947 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:02.362901    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:02.700161 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:02.700112    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:02.889783 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:02.889739    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:02.995445 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:02.995339    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:03.074326 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.074285    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-b47cd6d9b-bxz77"]
Sep 16 18:47:03.077906 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.077871    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.081174 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.081140    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:47:03.081174 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.081140    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:47:03.081393 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.081211    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:47:03.081393 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.081140    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-e2k6dgo5oeqj4\""
Sep 16 18:47:03.082699 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.082675    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:47:03.082859 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.082747    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-f89vg\""
Sep 16 18:47:03.087411 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.087379    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b47cd6d9b-bxz77"]
Sep 16 18:47:03.204773 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.204564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989" event={"ID":"306b52a5-f1ec-4522-bd1d-c5fb0430f230","Type":"ContainerStarted","Data":"09698f7b8facc0a168245473925a6b82f56c499c38941f213a4f7a5d87639c10"}
Sep 16 18:47:03.219087 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.219043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-client-ca-bundle\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.219344 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.219128    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-audit-log\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.219344 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.219178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-metrics-server-audit-profiles\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.219344 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.219208    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.219344 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.219242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-secret-metrics-server-tls\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.219344 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.219324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjntx\" (UniqueName: \"kubernetes.io/projected/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-kube-api-access-qjntx\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.219766 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.219414    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-secret-metrics-server-client-certs\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.320393 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.320352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-metrics-server-audit-profiles\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.320393 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.320403    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.320724 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.320437    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-secret-metrics-server-tls\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.320724 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.320475    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjntx\" (UniqueName: \"kubernetes.io/projected/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-kube-api-access-qjntx\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.320724 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.320503    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-secret-metrics-server-client-certs\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.320724 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.320538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-client-ca-bundle\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.320724 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.320619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-audit-log\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.321613 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.321165    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-audit-log\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.322331 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.322301    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.322449 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.322340    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-metrics-server-audit-profiles\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.324361 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.324333    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-secret-metrics-server-tls\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.325895 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.325872    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-secret-metrics-server-client-certs\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.328309 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.328289    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-client-ca-bundle\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.334335 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.334304    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjntx\" (UniqueName: \"kubernetes.io/projected/421dee84-b8ca-4684-bc2a-5ef7b6b3adf8-kube-api-access-qjntx\") pod \"metrics-server-b47cd6d9b-bxz77\" (UID: \"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8\") " pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.391425 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.391363    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:03.445119 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.445036    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:03.544118 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.544080    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:03.644174 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.644080    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:03.694397 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.694365    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b47cd6d9b-bxz77"]
Sep 16 18:47:03.797932 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:47:03.797894    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421dee84_b8ca_4684_bc2a_5ef7b6b3adf8.slice/crio-770c4816679c57b31fa9b6680769136ecfa7bf3dca6a0d14f8418866a28ab907 WatchSource:0}: Error finding container 770c4816679c57b31fa9b6680769136ecfa7bf3dca6a0d14f8418866a28ab907: Status 404 returned error can't find the container with id 770c4816679c57b31fa9b6680769136ecfa7bf3dca6a0d14f8418866a28ab907
Sep 16 18:47:03.897135 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:03.897015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:04.095875 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:04.095829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:04.211491 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:04.211387    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77" event={"ID":"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8","Type":"ContainerStarted","Data":"770c4816679c57b31fa9b6680769136ecfa7bf3dca6a0d14f8418866a28ab907"}
Sep 16 18:47:04.286046 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:04.285999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:04.386681 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:04.386654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:04.518828 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:04.518798    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:04.621280 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:04.621215    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:04.878028 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:04.877994    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:05.030618 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:05.030557    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:05.130601 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:05.130535    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:05.219736 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:05.219682    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989" event={"ID":"306b52a5-f1ec-4522-bd1d-c5fb0430f230","Type":"ContainerStarted","Data":"71a8d38a0aedaf13477e470baea3f9bc4ed36a3e9e57c956f6265fc0b6f1e83b"}
Sep 16 18:47:05.220251 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:05.219781    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989" event={"ID":"306b52a5-f1ec-4522-bd1d-c5fb0430f230","Type":"ContainerStarted","Data":"aebc98f7e6f8fe8a261295496acca9e8a112581b9d84ac4064fed88b5fcb623f"}
Sep 16 18:47:05.220251 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:05.219813    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989" event={"ID":"306b52a5-f1ec-4522-bd1d-c5fb0430f230","Type":"ContainerStarted","Data":"addc17bf59e5b215856c4f8ab342735f0d8b80d25d795d4b300662bd7eac5d05"}
Sep 16 18:47:05.223772 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:05.223723    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerStarted","Data":"7c890d9aa81678eb8f8a89b1b0eb697cd92e9629228566931b34e426fdf2ee7e"}
Sep 16 18:47:05.223964 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:05.223810    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerStarted","Data":"0691cc246f710ac4a2ef54d1f2bb8e89fe067e4463e52d3943a2b097e20ae6d7"}
Sep 16 18:47:05.223964 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:05.223843    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerStarted","Data":"fe5a5cf05ad8d567904ba5484391bd242bb616440d1db7f7f0509c92a4ece87d"}
Sep 16 18:47:05.223964 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:05.223884    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerStarted","Data":"eb98870bbd41e4b22e5ebdd37ba1c0c13580f4cb6956a3fdde36baeda06a57d6"}
Sep 16 18:47:05.223964 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:05.223916    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerStarted","Data":"09f41b13f7b2b0b5ca90174c216b6c3b9c94f9b8bbaedabe107c2a12583c5195"}
Sep 16 18:47:06.232497 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:06.232450    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989" event={"ID":"306b52a5-f1ec-4522-bd1d-c5fb0430f230","Type":"ContainerStarted","Data":"8af9a6bd674bad84b213ff8c38a80d965a75a2273d63a8e210e6e924e4703d2a"}
Sep 16 18:47:06.232998 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:06.232741    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:06.232998 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:06.232837    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989" event={"ID":"306b52a5-f1ec-4522-bd1d-c5fb0430f230","Type":"ContainerStarted","Data":"2bb72bab59b1a2d942103c349c40de7c84b18d3fb31200b9789910a595cba901"}
Sep 16 18:47:06.232998 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:06.232887    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989" event={"ID":"306b52a5-f1ec-4522-bd1d-c5fb0430f230","Type":"ContainerStarted","Data":"f9741554837fc8185d343b5f13982254aefb532a0c9940b829fe03ba7c31a552"}
Sep 16 18:47:06.234182 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:06.234147    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77" event={"ID":"421dee84-b8ca-4684-bc2a-5ef7b6b3adf8","Type":"ContainerStarted","Data":"f67d0d123f3158d15d02f57df55fae9138a88cd7f5e5cac13384b717d172fe08"}
Sep 16 18:47:06.238077 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:06.238049    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerStarted","Data":"7d3e943cc86a6a86969774cdc104c202efa98f608d96a0d5a1fe477fd3003eed"}
Sep 16 18:47:06.258860 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:06.258800    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989" podStartSLOduration=1.888988779 podStartE2EDuration="5.258774368s" totalImagesPullingTime="3.369785589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:06.257349266 +0000 UTC m=+84.087957834" watchObservedRunningTime="2026-09-16 18:47:06.258774368 +0000 UTC m=+84.089382912"
Sep 16 18:47:06.275693 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:06.275617    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77" podStartSLOduration=1.5088895039999999 podStartE2EDuration="3.27559609s" totalImagesPullingTime="1.766706586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:06.274844452 +0000 UTC m=+84.105453001" watchObservedRunningTime="2026-09-16 18:47:06.27559609 +0000 UTC m=+84.106204626"
Sep 16 18:47:06.307083 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:06.307023    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.813495888 podStartE2EDuration="7.307001905s" totalImagesPullingTime="4.474127295s" totalInitContainerRuntime="19.378722ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:06.304554238 +0000 UTC m=+84.135162781" watchObservedRunningTime="2026-09-16 18:47:06.307001905 +0000 UTC m=+84.137610451"
Sep 16 18:47:07.216972 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:07.216903    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:07.327846 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:07.327803    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:09.866756 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:09.866718    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:09.967411 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:09.967369    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:12.248785 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:12.248751    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6fd4d644d6-42989"
Sep 16 18:47:13.148821 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.148784    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-99b67cb9-kcqjd"
Sep 16 18:47:13.292585 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.292549    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:13.392875 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.392839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:13.452974 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.452885    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:13.524001 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.523965    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-868c9d85c-d6tfb"]
Sep 16 18:47:13.530993 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.530967    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.534153 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.534120    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:47:13.534365 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.534342    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:47:13.534477 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.534207    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:47:13.534477 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.534225    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pnm2k\""
Sep 16 18:47:13.534477 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.534263    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:47:13.534658 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.534223    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:47:13.535209 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.535193    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:47:13.535312 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.535297    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:47:13.538878 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.538859    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868c9d85c-d6tfb"]
Sep 16 18:47:13.615611 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.615539    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-config\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.615837 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.615660    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-oauth-config\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.615837 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.615705    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-service-ca\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.615837 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.615790    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-serving-cert\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.615837 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.615829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c5qg\" (UniqueName: \"kubernetes.io/projected/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-kube-api-access-9c5qg\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.616034 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.615901    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-oauth-serving-cert\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.716833 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.716733    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-config\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.716833 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.716780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-oauth-config\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.717056 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.716847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-service-ca\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.717056 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.716918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-serving-cert\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.717056 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.716948    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c5qg\" (UniqueName: \"kubernetes.io/projected/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-kube-api-access-9c5qg\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.717056 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.716997    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-oauth-serving-cert\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.717717 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.717694    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-config\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.717831 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.717696    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-service-ca\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.718014 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.717993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-oauth-serving-cert\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.719549 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.719527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-serving-cert\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.719656 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.719557    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-oauth-config\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.725564 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.725531    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c5qg\" (UniqueName: \"kubernetes.io/projected/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-kube-api-access-9c5qg\") pod \"console-868c9d85c-d6tfb\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") " pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:13.841778 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:13.841727    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:14.117378 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:14.117209    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868c9d85c-d6tfb"]
Sep 16 18:47:14.120665 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:47:14.120628    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fbfd9c_e00b_49b8_af5f_22495d9b3978.slice/crio-5d773ca66dc07a38b862bb6b58fcb6513c67995ab6c9fd8940f9248997ef0bad WatchSource:0}: Error finding container 5d773ca66dc07a38b862bb6b58fcb6513c67995ab6c9fd8940f9248997ef0bad: Status 404 returned error can't find the container with id 5d773ca66dc07a38b862bb6b58fcb6513c67995ab6c9fd8940f9248997ef0bad
Sep 16 18:47:14.273008 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:14.272957    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868c9d85c-d6tfb" event={"ID":"b4fbfd9c-e00b-49b8-af5f-22495d9b3978","Type":"ContainerStarted","Data":"5d773ca66dc07a38b862bb6b58fcb6513c67995ab6c9fd8940f9248997ef0bad"}
Sep 16 18:47:16.897742 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:16.897701    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:16.957810 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:16.957774    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:17.288503 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:17.288462    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868c9d85c-d6tfb" event={"ID":"b4fbfd9c-e00b-49b8-af5f-22495d9b3978","Type":"ContainerStarted","Data":"4f1043ea6b62d9b72fb6c1288253e09a7207a31b5b6891c04c6acfa2f1fa5967"}
Sep 16 18:47:17.307246 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:17.307186    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-868c9d85c-d6tfb" podStartSLOduration=1.673339964 podStartE2EDuration="4.307167338s" totalImagesPullingTime="2.633827374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:17.305736626 +0000 UTC m=+95.136345184" watchObservedRunningTime="2026-09-16 18:47:17.307167338 +0000 UTC m=+95.137775882"
Sep 16 18:47:17.505758 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:17.505720    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:17.573770 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:17.573664    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:19.123217 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:19.123187    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:19.180727 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:19.180692    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:21.519064 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:21.519023    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:21.616338 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:21.616298    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:22.902105 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:22.902039    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-df5dbc8fc-h5l92"]
Sep 16 18:47:22.907886 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:22.907863    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:22.917656 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:22.917629    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:47:22.919758 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:22.919729    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-df5dbc8fc-h5l92"]
Sep 16 18:47:22.997356 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:22.997307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78ef053c-664c-4555-98be-fbf381dd65e2-console-serving-cert\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:22.997356 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:22.997363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-service-ca\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:22.997649 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:22.997393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-trusted-ca-bundle\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:22.997649 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:22.997418    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78ef053c-664c-4555-98be-fbf381dd65e2-console-oauth-config\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:22.997649 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:22.997444    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-console-config\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:22.997649 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:22.997506    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-oauth-serving-cert\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:22.997649 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:22.997540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6w2n\" (UniqueName: \"kubernetes.io/projected/78ef053c-664c-4555-98be-fbf381dd65e2-kube-api-access-c6w2n\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.098143 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.098093    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-console-config\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.098143 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.098147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-oauth-serving-cert\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.098419 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.098170    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6w2n\" (UniqueName: \"kubernetes.io/projected/78ef053c-664c-4555-98be-fbf381dd65e2-kube-api-access-c6w2n\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.098419 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.098209    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78ef053c-664c-4555-98be-fbf381dd65e2-console-serving-cert\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.098419 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.098254    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-service-ca\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.098419 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.098281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-trusted-ca-bundle\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.098419 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.098315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78ef053c-664c-4555-98be-fbf381dd65e2-console-oauth-config\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.099109 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.099081    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-oauth-serving-cert\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.099227 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.099104    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-console-config\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.099227 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.099081    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-service-ca\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.099299 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.099236    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-trusted-ca-bundle\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.101653 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.101626    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78ef053c-664c-4555-98be-fbf381dd65e2-console-serving-cert\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.101771 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.101748    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78ef053c-664c-4555-98be-fbf381dd65e2-console-oauth-config\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.107829 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.107805    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6w2n\" (UniqueName: \"kubernetes.io/projected/78ef053c-664c-4555-98be-fbf381dd65e2-kube-api-access-c6w2n\") pod \"console-df5dbc8fc-h5l92\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") " pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.218858 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.218732    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:23.251311 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.251269    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:23.320702 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.320645    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:23.392468 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.392428    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:23.392675 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.392502    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:23.497270 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.497126    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-df5dbc8fc-h5l92"]
Sep 16 18:47:23.500365 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:47:23.500333    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ef053c_664c_4555_98be_fbf381dd65e2.slice/crio-1f1ef029f640b2dba3a50ac809d5fffc5d790dc4ac32de4b48479d2db23b5a9d WatchSource:0}: Error finding container 1f1ef029f640b2dba3a50ac809d5fffc5d790dc4ac32de4b48479d2db23b5a9d: Status 404 returned error can't find the container with id 1f1ef029f640b2dba3a50ac809d5fffc5d790dc4ac32de4b48479d2db23b5a9d
Sep 16 18:47:23.842055 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.842013    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:23.842272 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.842097    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:23.847990 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:23.847961    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:24.320026 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:24.319972    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df5dbc8fc-h5l92" event={"ID":"78ef053c-664c-4555-98be-fbf381dd65e2","Type":"ContainerStarted","Data":"dc55106ba80ec3e616d65380ee1f8cebafc81d8736beb63862fc16884f7b67c7"}
Sep 16 18:47:24.320605 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:24.320092    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df5dbc8fc-h5l92" event={"ID":"78ef053c-664c-4555-98be-fbf381dd65e2","Type":"ContainerStarted","Data":"1f1ef029f640b2dba3a50ac809d5fffc5d790dc4ac32de4b48479d2db23b5a9d"}
Sep 16 18:47:24.324391 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:24.324365    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:24.339547 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:24.339482    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-df5dbc8fc-h5l92" podStartSLOduration=2.339464419 podStartE2EDuration="2.339464419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:24.338794382 +0000 UTC m=+102.169402926" watchObservedRunningTime="2026-09-16 18:47:24.339464419 +0000 UTC m=+102.170073063"
Sep 16 18:47:29.178950 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:29.178914    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jsfns"
Sep 16 18:47:30.514972 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:30.514929    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:30.578445 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:30.578408    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:31.899612 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:31.899554    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:32.000549 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:32.000512    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:33.219195 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:33.219157    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:33.219679 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:33.219227    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:33.224406 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:33.224381    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:33.364514 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:33.364482    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:47:33.412465 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:33.412432    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-868c9d85c-d6tfb"]
Sep 16 18:47:33.739920 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:33.739878    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:33.797956 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:33.797918    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:35.367759 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:35.367726    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:35.434500 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:35.434474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:41.078245 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:41.078191    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:41.180982 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:41.180936    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:43.396949 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:43.396914    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:43.401126 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:43.401100    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-b47cd6d9b-bxz77"
Sep 16 18:47:53.313683 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:53.313640    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:53.411131 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:53.411093    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:53.511404 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:53.511366    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:58.437397 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.437340    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-868c9d85c-d6tfb" podUID="b4fbfd9c-e00b-49b8-af5f-22495d9b3978" containerName="console" containerID="cri-o://4f1043ea6b62d9b72fb6c1288253e09a7207a31b5b6891c04c6acfa2f1fa5967" gracePeriod=15
Sep 16 18:47:58.689476 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.689386    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:58.824546 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.824501    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c5qg\" (UniqueName: \"kubernetes.io/projected/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-kube-api-access-9c5qg\") pod \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") "
Sep 16 18:47:58.824546 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.824547    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-service-ca\") pod \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") "
Sep 16 18:47:58.824857 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.824595    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-oauth-config\") pod \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") "
Sep 16 18:47:58.824857 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.824735    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-oauth-serving-cert\") pod \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") "
Sep 16 18:47:58.824857 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.824794    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-config\") pod \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") "
Sep 16 18:47:58.824857 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.824859    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-serving-cert\") pod \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\" (UID: \"b4fbfd9c-e00b-49b8-af5f-22495d9b3978\") "
Sep 16 18:47:58.825152 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.825040    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-service-ca" (OuterVolumeSpecName: "service-ca") pod "b4fbfd9c-e00b-49b8-af5f-22495d9b3978" (UID: "b4fbfd9c-e00b-49b8-af5f-22495d9b3978"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:58.825152 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.825149    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b4fbfd9c-e00b-49b8-af5f-22495d9b3978" (UID: "b4fbfd9c-e00b-49b8-af5f-22495d9b3978"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:58.825313 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.825284    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-config" (OuterVolumeSpecName: "console-config") pod "b4fbfd9c-e00b-49b8-af5f-22495d9b3978" (UID: "b4fbfd9c-e00b-49b8-af5f-22495d9b3978"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:58.827277 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.827242    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b4fbfd9c-e00b-49b8-af5f-22495d9b3978" (UID: "b4fbfd9c-e00b-49b8-af5f-22495d9b3978"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:58.827277 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.827258    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b4fbfd9c-e00b-49b8-af5f-22495d9b3978" (UID: "b4fbfd9c-e00b-49b8-af5f-22495d9b3978"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:58.827453 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.827306    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-kube-api-access-9c5qg" (OuterVolumeSpecName: "kube-api-access-9c5qg") pod "b4fbfd9c-e00b-49b8-af5f-22495d9b3978" (UID: "b4fbfd9c-e00b-49b8-af5f-22495d9b3978"). InnerVolumeSpecName "kube-api-access-9c5qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:58.926592 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.926520    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-serving-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:47:58.926592 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.926586    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c5qg\" (UniqueName: \"kubernetes.io/projected/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-kube-api-access-9c5qg\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:47:58.926592 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.926602    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-service-ca\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:47:58.926592 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.926611    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-oauth-config\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:47:58.926592 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.926620    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-oauth-serving-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:47:58.926911 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:58.926629    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4fbfd9c-e00b-49b8-af5f-22495d9b3978-console-config\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:47:59.464159 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:59.464112    2582 generic.go:356] "Generic (PLEG): container finished" podID="b4fbfd9c-e00b-49b8-af5f-22495d9b3978" containerID="4f1043ea6b62d9b72fb6c1288253e09a7207a31b5b6891c04c6acfa2f1fa5967" exitCode=0
Sep 16 18:47:59.464648 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:59.464166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868c9d85c-d6tfb" event={"ID":"b4fbfd9c-e00b-49b8-af5f-22495d9b3978","Type":"ContainerDied","Data":"4f1043ea6b62d9b72fb6c1288253e09a7207a31b5b6891c04c6acfa2f1fa5967"}
Sep 16 18:47:59.464648 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:59.464186    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868c9d85c-d6tfb"
Sep 16 18:47:59.464648 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:59.464260    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868c9d85c-d6tfb" event={"ID":"b4fbfd9c-e00b-49b8-af5f-22495d9b3978","Type":"ContainerDied","Data":"5d773ca66dc07a38b862bb6b58fcb6513c67995ab6c9fd8940f9248997ef0bad"}
Sep 16 18:47:59.464648 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:59.464285    2582 scope.go:117] "RemoveContainer" containerID="4f1043ea6b62d9b72fb6c1288253e09a7207a31b5b6891c04c6acfa2f1fa5967"
Sep 16 18:47:59.473857 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:59.473832    2582 scope.go:117] "RemoveContainer" containerID="4f1043ea6b62d9b72fb6c1288253e09a7207a31b5b6891c04c6acfa2f1fa5967"
Sep 16 18:47:59.474190 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:47:59.474169    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1043ea6b62d9b72fb6c1288253e09a7207a31b5b6891c04c6acfa2f1fa5967\": container with ID starting with 4f1043ea6b62d9b72fb6c1288253e09a7207a31b5b6891c04c6acfa2f1fa5967 not found: ID does not exist" containerID="4f1043ea6b62d9b72fb6c1288253e09a7207a31b5b6891c04c6acfa2f1fa5967"
Sep 16 18:47:59.474265 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:59.474205    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1043ea6b62d9b72fb6c1288253e09a7207a31b5b6891c04c6acfa2f1fa5967"} err="failed to get container status \"4f1043ea6b62d9b72fb6c1288253e09a7207a31b5b6891c04c6acfa2f1fa5967\": rpc error: code = NotFound desc = could not find container \"4f1043ea6b62d9b72fb6c1288253e09a7207a31b5b6891c04c6acfa2f1fa5967\": container with ID starting with 4f1043ea6b62d9b72fb6c1288253e09a7207a31b5b6891c04c6acfa2f1fa5967 not found: ID does not exist"
Sep 16 18:47:59.487667 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:59.487549    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-868c9d85c-d6tfb"]
Sep 16 18:47:59.491587 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:47:59.491472    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-868c9d85c-d6tfb"]
Sep 16 18:48:00.703287 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:00.703233    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4fbfd9c-e00b-49b8-af5f-22495d9b3978" path="/var/lib/kubelet/pods/b4fbfd9c-e00b-49b8-af5f-22495d9b3978/volumes"
Sep 16 18:48:17.889412 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:17.889380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:18.191828 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:18.191754    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:18.898393 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:18.898365    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:18.898788 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:18.898646    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="alertmanager" containerID="cri-o://09f41b13f7b2b0b5ca90174c216b6c3b9c94f9b8bbaedabe107c2a12583c5195" gracePeriod=120
Sep 16 18:48:18.898788 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:18.898716    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="kube-rbac-proxy-metric" containerID="cri-o://7c890d9aa81678eb8f8a89b1b0eb697cd92e9629228566931b34e426fdf2ee7e" gracePeriod=120
Sep 16 18:48:18.898788 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:18.898769    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="kube-rbac-proxy" containerID="cri-o://0691cc246f710ac4a2ef54d1f2bb8e89fe067e4463e52d3943a2b097e20ae6d7" gracePeriod=120
Sep 16 18:48:18.898951 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:18.898775    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="config-reloader" containerID="cri-o://eb98870bbd41e4b22e5ebdd37ba1c0c13580f4cb6956a3fdde36baeda06a57d6" gracePeriod=120
Sep 16 18:48:18.898951 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:18.898796    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="prom-label-proxy" containerID="cri-o://7d3e943cc86a6a86969774cdc104c202efa98f608d96a0d5a1fe477fd3003eed" gracePeriod=120
Sep 16 18:48:18.898951 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:18.898718    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="kube-rbac-proxy-web" containerID="cri-o://fe5a5cf05ad8d567904ba5484391bd242bb616440d1db7f7f0509c92a4ece87d" gracePeriod=120
Sep 16 18:48:19.554124 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:19.554096    2582 generic.go:356] "Generic (PLEG): container finished" podID="c448a1cc-5061-4290-99d4-39c846818630" containerID="7d3e943cc86a6a86969774cdc104c202efa98f608d96a0d5a1fe477fd3003eed" exitCode=0
Sep 16 18:48:19.554124 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:19.554119    2582 generic.go:356] "Generic (PLEG): container finished" podID="c448a1cc-5061-4290-99d4-39c846818630" containerID="7c890d9aa81678eb8f8a89b1b0eb697cd92e9629228566931b34e426fdf2ee7e" exitCode=0
Sep 16 18:48:19.554124 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:19.554126    2582 generic.go:356] "Generic (PLEG): container finished" podID="c448a1cc-5061-4290-99d4-39c846818630" containerID="0691cc246f710ac4a2ef54d1f2bb8e89fe067e4463e52d3943a2b097e20ae6d7" exitCode=0
Sep 16 18:48:19.554292 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:19.554131    2582 generic.go:356] "Generic (PLEG): container finished" podID="c448a1cc-5061-4290-99d4-39c846818630" containerID="eb98870bbd41e4b22e5ebdd37ba1c0c13580f4cb6956a3fdde36baeda06a57d6" exitCode=0
Sep 16 18:48:19.554292 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:19.554136    2582 generic.go:356] "Generic (PLEG): container finished" podID="c448a1cc-5061-4290-99d4-39c846818630" containerID="09f41b13f7b2b0b5ca90174c216b6c3b9c94f9b8bbaedabe107c2a12583c5195" exitCode=0
Sep 16 18:48:19.554292 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:19.554171    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerDied","Data":"7d3e943cc86a6a86969774cdc104c202efa98f608d96a0d5a1fe477fd3003eed"}
Sep 16 18:48:19.554292 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:19.554229    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerDied","Data":"7c890d9aa81678eb8f8a89b1b0eb697cd92e9629228566931b34e426fdf2ee7e"}
Sep 16 18:48:19.554292 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:19.554256    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerDied","Data":"0691cc246f710ac4a2ef54d1f2bb8e89fe067e4463e52d3943a2b097e20ae6d7"}
Sep 16 18:48:19.554292 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:19.554281    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerDied","Data":"eb98870bbd41e4b22e5ebdd37ba1c0c13580f4cb6956a3fdde36baeda06a57d6"}
Sep 16 18:48:19.554459 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:19.554304    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerDied","Data":"09f41b13f7b2b0b5ca90174c216b6c3b9c94f9b8bbaedabe107c2a12583c5195"}
Sep 16 18:48:20.151146 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.151125    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.198302 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.198277    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5x27\" (UniqueName: \"kubernetes.io/projected/c448a1cc-5061-4290-99d4-39c846818630-kube-api-access-x5x27\") pod \"c448a1cc-5061-4290-99d4-39c846818630\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") "
Sep 16 18:48:20.198430 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.198315    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-cluster-tls-config\") pod \"c448a1cc-5061-4290-99d4-39c846818630\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") "
Sep 16 18:48:20.198430 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.198339    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy-web\") pod \"c448a1cc-5061-4290-99d4-39c846818630\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") "
Sep 16 18:48:20.198430 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.198362    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-web-config\") pod \"c448a1cc-5061-4290-99d4-39c846818630\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") "
Sep 16 18:48:20.198430 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.198390    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy\") pod \"c448a1cc-5061-4290-99d4-39c846818630\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") "
Sep 16 18:48:20.198430 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.198413    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-metrics-client-ca\") pod \"c448a1cc-5061-4290-99d4-39c846818630\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") "
Sep 16 18:48:20.198722 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.198471    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-config-volume\") pod \"c448a1cc-5061-4290-99d4-39c846818630\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") "
Sep 16 18:48:20.198722 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.198501    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-trusted-ca-bundle\") pod \"c448a1cc-5061-4290-99d4-39c846818630\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") "
Sep 16 18:48:20.198722 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.198541    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c448a1cc-5061-4290-99d4-39c846818630-tls-assets\") pod \"c448a1cc-5061-4290-99d4-39c846818630\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") "
Sep 16 18:48:20.198722 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.198597    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c448a1cc-5061-4290-99d4-39c846818630\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") "
Sep 16 18:48:20.198722 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.198637    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c448a1cc-5061-4290-99d4-39c846818630-config-out\") pod \"c448a1cc-5061-4290-99d4-39c846818630\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") "
Sep 16 18:48:20.198722 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.198698    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-main-tls\") pod \"c448a1cc-5061-4290-99d4-39c846818630\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") "
Sep 16 18:48:20.199017 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.198730    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-main-db\") pod \"c448a1cc-5061-4290-99d4-39c846818630\" (UID: \"c448a1cc-5061-4290-99d4-39c846818630\") "
Sep 16 18:48:20.199017 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.198978    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c448a1cc-5061-4290-99d4-39c846818630" (UID: "c448a1cc-5061-4290-99d4-39c846818630"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:20.199119 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.199039    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c448a1cc-5061-4290-99d4-39c846818630" (UID: "c448a1cc-5061-4290-99d4-39c846818630"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:20.199834 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.199778    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c448a1cc-5061-4290-99d4-39c846818630" (UID: "c448a1cc-5061-4290-99d4-39c846818630"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:20.202354 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.202317    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-config-volume" (OuterVolumeSpecName: "config-volume") pod "c448a1cc-5061-4290-99d4-39c846818630" (UID: "c448a1cc-5061-4290-99d4-39c846818630"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:20.202464 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.202367    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c448a1cc-5061-4290-99d4-39c846818630" (UID: "c448a1cc-5061-4290-99d4-39c846818630"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:20.202740 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.202705    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c448a1cc-5061-4290-99d4-39c846818630-config-out" (OuterVolumeSpecName: "config-out") pod "c448a1cc-5061-4290-99d4-39c846818630" (UID: "c448a1cc-5061-4290-99d4-39c846818630"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:20.202843 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.202752    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c448a1cc-5061-4290-99d4-39c846818630-kube-api-access-x5x27" (OuterVolumeSpecName: "kube-api-access-x5x27") pod "c448a1cc-5061-4290-99d4-39c846818630" (UID: "c448a1cc-5061-4290-99d4-39c846818630"). InnerVolumeSpecName "kube-api-access-x5x27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:20.203504 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.203467    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c448a1cc-5061-4290-99d4-39c846818630-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c448a1cc-5061-4290-99d4-39c846818630" (UID: "c448a1cc-5061-4290-99d4-39c846818630"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:20.203751 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.203714    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c448a1cc-5061-4290-99d4-39c846818630" (UID: "c448a1cc-5061-4290-99d4-39c846818630"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:20.204131 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.204113    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c448a1cc-5061-4290-99d4-39c846818630" (UID: "c448a1cc-5061-4290-99d4-39c846818630"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:20.204397 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.204376    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c448a1cc-5061-4290-99d4-39c846818630" (UID: "c448a1cc-5061-4290-99d4-39c846818630"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:20.207734 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.207688    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c448a1cc-5061-4290-99d4-39c846818630" (UID: "c448a1cc-5061-4290-99d4-39c846818630"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:20.213450 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.213388    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-web-config" (OuterVolumeSpecName: "web-config") pod "c448a1cc-5061-4290-99d4-39c846818630" (UID: "c448a1cc-5061-4290-99d4-39c846818630"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:20.299879 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.299855    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.299879 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.299876    2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-web-config\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.299994 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.299887    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.299994 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.299896    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-metrics-client-ca\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.299994 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.299904    2582 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-config-volume\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.299994 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.299912    2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.299994 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.299921    2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c448a1cc-5061-4290-99d4-39c846818630-tls-assets\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.299994 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.299930    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.299994 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.299938    2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c448a1cc-5061-4290-99d4-39c846818630-config-out\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.299994 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.299945    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-secret-alertmanager-main-tls\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.299994 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.299953    2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c448a1cc-5061-4290-99d4-39c846818630-alertmanager-main-db\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.299994 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.299962    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5x27\" (UniqueName: \"kubernetes.io/projected/c448a1cc-5061-4290-99d4-39c846818630-kube-api-access-x5x27\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.299994 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.299970    2582 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c448a1cc-5061-4290-99d4-39c846818630-cluster-tls-config\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:48:20.561048 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.561018    2582 generic.go:356] "Generic (PLEG): container finished" podID="c448a1cc-5061-4290-99d4-39c846818630" containerID="fe5a5cf05ad8d567904ba5484391bd242bb616440d1db7f7f0509c92a4ece87d" exitCode=0
Sep 16 18:48:20.561191 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.561062    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerDied","Data":"fe5a5cf05ad8d567904ba5484391bd242bb616440d1db7f7f0509c92a4ece87d"}
Sep 16 18:48:20.561191 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.561109    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.561191 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.561119    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c448a1cc-5061-4290-99d4-39c846818630","Type":"ContainerDied","Data":"b26185e1d284f40ac56a1badcc17e36403237904f18ea8c9f2712fc5fd0a253a"}
Sep 16 18:48:20.561191 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.561155    2582 scope.go:117] "RemoveContainer" containerID="7d3e943cc86a6a86969774cdc104c202efa98f608d96a0d5a1fe477fd3003eed"
Sep 16 18:48:20.569327 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.569308    2582 scope.go:117] "RemoveContainer" containerID="7c890d9aa81678eb8f8a89b1b0eb697cd92e9629228566931b34e426fdf2ee7e"
Sep 16 18:48:20.576047 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.576031    2582 scope.go:117] "RemoveContainer" containerID="0691cc246f710ac4a2ef54d1f2bb8e89fe067e4463e52d3943a2b097e20ae6d7"
Sep 16 18:48:20.582439 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.582422    2582 scope.go:117] "RemoveContainer" containerID="fe5a5cf05ad8d567904ba5484391bd242bb616440d1db7f7f0509c92a4ece87d"
Sep 16 18:48:20.585806 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.585786    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:20.589725 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.589629    2582 scope.go:117] "RemoveContainer" containerID="eb98870bbd41e4b22e5ebdd37ba1c0c13580f4cb6956a3fdde36baeda06a57d6"
Sep 16 18:48:20.590918 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.590902    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:20.596486 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.596465    2582 scope.go:117] "RemoveContainer" containerID="09f41b13f7b2b0b5ca90174c216b6c3b9c94f9b8bbaedabe107c2a12583c5195"
Sep 16 18:48:20.603073 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.603056    2582 scope.go:117] "RemoveContainer" containerID="619f57411785059460784b20c17188adea750b98387894e2c258c06ebbe6061a"
Sep 16 18:48:20.609909 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.609878    2582 scope.go:117] "RemoveContainer" containerID="7d3e943cc86a6a86969774cdc104c202efa98f608d96a0d5a1fe477fd3003eed"
Sep 16 18:48:20.610154 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:48:20.610137    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d3e943cc86a6a86969774cdc104c202efa98f608d96a0d5a1fe477fd3003eed\": container with ID starting with 7d3e943cc86a6a86969774cdc104c202efa98f608d96a0d5a1fe477fd3003eed not found: ID does not exist" containerID="7d3e943cc86a6a86969774cdc104c202efa98f608d96a0d5a1fe477fd3003eed"
Sep 16 18:48:20.610205 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.610164    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3e943cc86a6a86969774cdc104c202efa98f608d96a0d5a1fe477fd3003eed"} err="failed to get container status \"7d3e943cc86a6a86969774cdc104c202efa98f608d96a0d5a1fe477fd3003eed\": rpc error: code = NotFound desc = could not find container \"7d3e943cc86a6a86969774cdc104c202efa98f608d96a0d5a1fe477fd3003eed\": container with ID starting with 7d3e943cc86a6a86969774cdc104c202efa98f608d96a0d5a1fe477fd3003eed not found: ID does not exist"
Sep 16 18:48:20.610205 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.610185    2582 scope.go:117] "RemoveContainer" containerID="7c890d9aa81678eb8f8a89b1b0eb697cd92e9629228566931b34e426fdf2ee7e"
Sep 16 18:48:20.610448 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:48:20.610430    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c890d9aa81678eb8f8a89b1b0eb697cd92e9629228566931b34e426fdf2ee7e\": container with ID starting with 7c890d9aa81678eb8f8a89b1b0eb697cd92e9629228566931b34e426fdf2ee7e not found: ID does not exist" containerID="7c890d9aa81678eb8f8a89b1b0eb697cd92e9629228566931b34e426fdf2ee7e"
Sep 16 18:48:20.610497 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.610453    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c890d9aa81678eb8f8a89b1b0eb697cd92e9629228566931b34e426fdf2ee7e"} err="failed to get container status \"7c890d9aa81678eb8f8a89b1b0eb697cd92e9629228566931b34e426fdf2ee7e\": rpc error: code = NotFound desc = could not find container \"7c890d9aa81678eb8f8a89b1b0eb697cd92e9629228566931b34e426fdf2ee7e\": container with ID starting with 7c890d9aa81678eb8f8a89b1b0eb697cd92e9629228566931b34e426fdf2ee7e not found: ID does not exist"
Sep 16 18:48:20.610497 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.610469    2582 scope.go:117] "RemoveContainer" containerID="0691cc246f710ac4a2ef54d1f2bb8e89fe067e4463e52d3943a2b097e20ae6d7"
Sep 16 18:48:20.610745 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:48:20.610729    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0691cc246f710ac4a2ef54d1f2bb8e89fe067e4463e52d3943a2b097e20ae6d7\": container with ID starting with 0691cc246f710ac4a2ef54d1f2bb8e89fe067e4463e52d3943a2b097e20ae6d7 not found: ID does not exist" containerID="0691cc246f710ac4a2ef54d1f2bb8e89fe067e4463e52d3943a2b097e20ae6d7"
Sep 16 18:48:20.610788 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.610749    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0691cc246f710ac4a2ef54d1f2bb8e89fe067e4463e52d3943a2b097e20ae6d7"} err="failed to get container status \"0691cc246f710ac4a2ef54d1f2bb8e89fe067e4463e52d3943a2b097e20ae6d7\": rpc error: code = NotFound desc = could not find container \"0691cc246f710ac4a2ef54d1f2bb8e89fe067e4463e52d3943a2b097e20ae6d7\": container with ID starting with 0691cc246f710ac4a2ef54d1f2bb8e89fe067e4463e52d3943a2b097e20ae6d7 not found: ID does not exist"
Sep 16 18:48:20.610788 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.610763    2582 scope.go:117] "RemoveContainer" containerID="fe5a5cf05ad8d567904ba5484391bd242bb616440d1db7f7f0509c92a4ece87d"
Sep 16 18:48:20.611016 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:48:20.611000    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5a5cf05ad8d567904ba5484391bd242bb616440d1db7f7f0509c92a4ece87d\": container with ID starting with fe5a5cf05ad8d567904ba5484391bd242bb616440d1db7f7f0509c92a4ece87d not found: ID does not exist" containerID="fe5a5cf05ad8d567904ba5484391bd242bb616440d1db7f7f0509c92a4ece87d"
Sep 16 18:48:20.611059 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.611019    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5a5cf05ad8d567904ba5484391bd242bb616440d1db7f7f0509c92a4ece87d"} err="failed to get container status \"fe5a5cf05ad8d567904ba5484391bd242bb616440d1db7f7f0509c92a4ece87d\": rpc error: code = NotFound desc = could not find container \"fe5a5cf05ad8d567904ba5484391bd242bb616440d1db7f7f0509c92a4ece87d\": container with ID starting with fe5a5cf05ad8d567904ba5484391bd242bb616440d1db7f7f0509c92a4ece87d not found: ID does not exist"
Sep 16 18:48:20.611059 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.611030    2582 scope.go:117] "RemoveContainer" containerID="eb98870bbd41e4b22e5ebdd37ba1c0c13580f4cb6956a3fdde36baeda06a57d6"
Sep 16 18:48:20.611260 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:48:20.611244    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb98870bbd41e4b22e5ebdd37ba1c0c13580f4cb6956a3fdde36baeda06a57d6\": container with ID starting with eb98870bbd41e4b22e5ebdd37ba1c0c13580f4cb6956a3fdde36baeda06a57d6 not found: ID does not exist" containerID="eb98870bbd41e4b22e5ebdd37ba1c0c13580f4cb6956a3fdde36baeda06a57d6"
Sep 16 18:48:20.611304 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.611265    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb98870bbd41e4b22e5ebdd37ba1c0c13580f4cb6956a3fdde36baeda06a57d6"} err="failed to get container status \"eb98870bbd41e4b22e5ebdd37ba1c0c13580f4cb6956a3fdde36baeda06a57d6\": rpc error: code = NotFound desc = could not find container \"eb98870bbd41e4b22e5ebdd37ba1c0c13580f4cb6956a3fdde36baeda06a57d6\": container with ID starting with eb98870bbd41e4b22e5ebdd37ba1c0c13580f4cb6956a3fdde36baeda06a57d6 not found: ID does not exist"
Sep 16 18:48:20.611304 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.611280    2582 scope.go:117] "RemoveContainer" containerID="09f41b13f7b2b0b5ca90174c216b6c3b9c94f9b8bbaedabe107c2a12583c5195"
Sep 16 18:48:20.611524 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:48:20.611508    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f41b13f7b2b0b5ca90174c216b6c3b9c94f9b8bbaedabe107c2a12583c5195\": container with ID starting with 09f41b13f7b2b0b5ca90174c216b6c3b9c94f9b8bbaedabe107c2a12583c5195 not found: ID does not exist" containerID="09f41b13f7b2b0b5ca90174c216b6c3b9c94f9b8bbaedabe107c2a12583c5195"
Sep 16 18:48:20.611584 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.611535    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f41b13f7b2b0b5ca90174c216b6c3b9c94f9b8bbaedabe107c2a12583c5195"} err="failed to get container status \"09f41b13f7b2b0b5ca90174c216b6c3b9c94f9b8bbaedabe107c2a12583c5195\": rpc error: code = NotFound desc = could not find container \"09f41b13f7b2b0b5ca90174c216b6c3b9c94f9b8bbaedabe107c2a12583c5195\": container with ID starting with 09f41b13f7b2b0b5ca90174c216b6c3b9c94f9b8bbaedabe107c2a12583c5195 not found: ID does not exist"
Sep 16 18:48:20.611584 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.611550    2582 scope.go:117] "RemoveContainer" containerID="619f57411785059460784b20c17188adea750b98387894e2c258c06ebbe6061a"
Sep 16 18:48:20.611799 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:48:20.611784    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619f57411785059460784b20c17188adea750b98387894e2c258c06ebbe6061a\": container with ID starting with 619f57411785059460784b20c17188adea750b98387894e2c258c06ebbe6061a not found: ID does not exist" containerID="619f57411785059460784b20c17188adea750b98387894e2c258c06ebbe6061a"
Sep 16 18:48:20.611845 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.611802    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619f57411785059460784b20c17188adea750b98387894e2c258c06ebbe6061a"} err="failed to get container status \"619f57411785059460784b20c17188adea750b98387894e2c258c06ebbe6061a\": rpc error: code = NotFound desc = could not find container \"619f57411785059460784b20c17188adea750b98387894e2c258c06ebbe6061a\": container with ID starting with 619f57411785059460784b20c17188adea750b98387894e2c258c06ebbe6061a not found: ID does not exist"
Sep 16 18:48:20.617901 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.617883    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:20.618128 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618117    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="prom-label-proxy"
Sep 16 18:48:20.618168 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618130    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="prom-label-proxy"
Sep 16 18:48:20.618168 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618142    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4fbfd9c-e00b-49b8-af5f-22495d9b3978" containerName="console"
Sep 16 18:48:20.618168 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618149    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4fbfd9c-e00b-49b8-af5f-22495d9b3978" containerName="console"
Sep 16 18:48:20.618168 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618159    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="alertmanager"
Sep 16 18:48:20.618168 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618165    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="alertmanager"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618175    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="config-reloader"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618180    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="config-reloader"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618186    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="kube-rbac-proxy-metric"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618191    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="kube-rbac-proxy-metric"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618200    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="kube-rbac-proxy-web"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618205    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="kube-rbac-proxy-web"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618213    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="init-config-reloader"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618218    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="init-config-reloader"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618223    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="kube-rbac-proxy"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618228    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="kube-rbac-proxy"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618271    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="alertmanager"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618278    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="kube-rbac-proxy-web"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618284    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="prom-label-proxy"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618289    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4fbfd9c-e00b-49b8-af5f-22495d9b3978" containerName="console"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618294    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="kube-rbac-proxy"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618300    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="config-reloader"
Sep 16 18:48:20.618315 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.618305    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c448a1cc-5061-4290-99d4-39c846818630" containerName="kube-rbac-proxy-metric"
Sep 16 18:48:20.621607 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.621593    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.624912 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.624891    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:48:20.624912 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.624905    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:48:20.625065 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.624934    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ddwbj\""
Sep 16 18:48:20.625065 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.624891    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:48:20.625065 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.624900    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:48:20.625065 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.624980    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:48:20.625728 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.625569    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:48:20.627053 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.626716    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:48:20.627053 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.626826    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:48:20.633957 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.633939    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:48:20.639048 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.639027    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:20.703077 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.703058    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c448a1cc-5061-4290-99d4-39c846818630" path="/var/lib/kubelet/pods/c448a1cc-5061-4290-99d4-39c846818630/volumes"
Sep 16 18:48:20.703468 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.703451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.703513 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.703479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-config-volume\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.703513 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.703495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.703629 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.703513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.703629 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.703564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.703629 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.703614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.703738 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.703632    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.703738 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.703648    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.703738 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.703668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.703738 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.703684    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-web-config\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.703738 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.703698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.703738 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.703732    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flp86\" (UniqueName: \"kubernetes.io/projected/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-kube-api-access-flp86\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.703933 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.703749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-config-out\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.804558 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.804525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.804558 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.804562    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-web-config\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.804798 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.804607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.804798 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.804640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flp86\" (UniqueName: \"kubernetes.io/projected/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-kube-api-access-flp86\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.804798 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.804659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-config-out\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.804798 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.804752    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.804798 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.804792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-config-volume\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.805034 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.804817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.805034 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.804847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.805034 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.804881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.805034 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.804908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.805034 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.804934    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.805034 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.804965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.805331 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.805060    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.807893 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.807866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-config-out\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.808030 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.807985    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-web-config\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.808030 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.807985    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.808153 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.808106    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.808153 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.808138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-config-volume\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.808278 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.808250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.808331 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.808274    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.808525 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.808508    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.808617 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.808527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.808680 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.808630    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.809910 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.809890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.813393 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.813350    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flp86\" (UniqueName: \"kubernetes.io/projected/636fd7d4-7686-4ea8-8d57-a534faa3c8ff-kube-api-access-flp86\") pod \"alertmanager-main-0\" (UID: \"636fd7d4-7686-4ea8-8d57-a534faa3c8ff\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:20.932860 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:20.932840    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:48:21.217976 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:21.217951    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:48:21.219602 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:48:21.219555    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636fd7d4_7686_4ea8_8d57_a534faa3c8ff.slice/crio-519d883e00a512b41e04ea272ac542e9547d98e1e6d2dbe6a0a5ec3c69b28059 WatchSource:0}: Error finding container 519d883e00a512b41e04ea272ac542e9547d98e1e6d2dbe6a0a5ec3c69b28059: Status 404 returned error can't find the container with id 519d883e00a512b41e04ea272ac542e9547d98e1e6d2dbe6a0a5ec3c69b28059
Sep 16 18:48:21.568328 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:21.568280    2582 generic.go:356] "Generic (PLEG): container finished" podID="636fd7d4-7686-4ea8-8d57-a534faa3c8ff" containerID="a4285c08e4f3e5aaef8197a4ca7aaedc709dafc6dda3536cf2ee3151ed6fe51d" exitCode=0
Sep 16 18:48:21.568519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:21.568349    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"636fd7d4-7686-4ea8-8d57-a534faa3c8ff","Type":"ContainerDied","Data":"a4285c08e4f3e5aaef8197a4ca7aaedc709dafc6dda3536cf2ee3151ed6fe51d"}
Sep 16 18:48:21.568519 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:21.568424    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"636fd7d4-7686-4ea8-8d57-a534faa3c8ff","Type":"ContainerStarted","Data":"519d883e00a512b41e04ea272ac542e9547d98e1e6d2dbe6a0a5ec3c69b28059"}
Sep 16 18:48:22.576101 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.576066    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"636fd7d4-7686-4ea8-8d57-a534faa3c8ff","Type":"ContainerStarted","Data":"8abadb866f561918354890eec7fcf0a25de00902a895fd76791480574d769754"}
Sep 16 18:48:22.576505 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.576132    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"636fd7d4-7686-4ea8-8d57-a534faa3c8ff","Type":"ContainerStarted","Data":"d57ce353bc4bb5bf0fefb81dd4f9aaf102dd3a4fb39409e94b067573ad08371f"}
Sep 16 18:48:22.576505 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.576162    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"636fd7d4-7686-4ea8-8d57-a534faa3c8ff","Type":"ContainerStarted","Data":"cd8166fed3d05a7ee8838998157da1706d21166bbed118fb8d6617032216c42f"}
Sep 16 18:48:22.576505 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.576184    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"636fd7d4-7686-4ea8-8d57-a534faa3c8ff","Type":"ContainerStarted","Data":"ebf4727a14159789cf7fc47d7fd7c04cdaff23efddeece4b85840bead267ceab"}
Sep 16 18:48:22.576505 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.576213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"636fd7d4-7686-4ea8-8d57-a534faa3c8ff","Type":"ContainerStarted","Data":"1516a736ae79da6f1af0e2e234922c85e93d64642b3ee78a1f9305effb164515"}
Sep 16 18:48:22.576505 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.576236    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"636fd7d4-7686-4ea8-8d57-a534faa3c8ff","Type":"ContainerStarted","Data":"23a98dc03179e7ca534073308b11e23a1baaefaa53fd7ece025fc56533cfac65"}
Sep 16 18:48:22.606494 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.606436    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.589990183 podStartE2EDuration="2.60635924s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.369057ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:22.603789698 +0000 UTC m=+160.434398243" watchObservedRunningTime="2026-09-16 18:48:22.60635924 +0000 UTC m=+160.436967784"
Sep 16 18:48:22.973310 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.973223    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"]
Sep 16 18:48:22.975920 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.975902    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:22.979089 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.979061    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:48:22.979241 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.979061    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:48:22.979490 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.979470    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:48:22.979569 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.979533    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:48:22.979569 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.979551    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:48:22.979678 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.979538    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-5t6ck\""
Sep 16 18:48:22.984840 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.984822    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:48:22.990134 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:22.990111    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"]
Sep 16 18:48:23.027767 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.027719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbfr9\" (UniqueName: \"kubernetes.io/projected/be1bfa15-8808-4400-b954-e13985b00908-kube-api-access-wbfr9\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.027767 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.027767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1bfa15-8808-4400-b954-e13985b00908-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.028023 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.027798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/be1bfa15-8808-4400-b954-e13985b00908-telemeter-client-tls\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.028023 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.027856    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/be1bfa15-8808-4400-b954-e13985b00908-federate-client-tls\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.028023 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.027892    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/be1bfa15-8808-4400-b954-e13985b00908-secret-telemeter-client\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.028023 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.027930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1bfa15-8808-4400-b954-e13985b00908-serving-certs-ca-bundle\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.028023 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.028019    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be1bfa15-8808-4400-b954-e13985b00908-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.028237 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.028046    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be1bfa15-8808-4400-b954-e13985b00908-metrics-client-ca\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.128907 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.128864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbfr9\" (UniqueName: \"kubernetes.io/projected/be1bfa15-8808-4400-b954-e13985b00908-kube-api-access-wbfr9\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.128907 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.128905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1bfa15-8808-4400-b954-e13985b00908-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.129199 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.128926    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/be1bfa15-8808-4400-b954-e13985b00908-telemeter-client-tls\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.129199 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.128951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/be1bfa15-8808-4400-b954-e13985b00908-federate-client-tls\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.129199 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.128981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/be1bfa15-8808-4400-b954-e13985b00908-secret-telemeter-client\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.129199 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.129016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1bfa15-8808-4400-b954-e13985b00908-serving-certs-ca-bundle\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.129199 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.129060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be1bfa15-8808-4400-b954-e13985b00908-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.129199 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.129085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be1bfa15-8808-4400-b954-e13985b00908-metrics-client-ca\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.130177 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.130143    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be1bfa15-8808-4400-b954-e13985b00908-metrics-client-ca\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.130345 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.130143    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1bfa15-8808-4400-b954-e13985b00908-serving-certs-ca-bundle\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.130345 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.130265    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1bfa15-8808-4400-b954-e13985b00908-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.131881 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.131857    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/be1bfa15-8808-4400-b954-e13985b00908-federate-client-tls\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.132024 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.132004    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/be1bfa15-8808-4400-b954-e13985b00908-telemeter-client-tls\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.132077 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.132064    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/be1bfa15-8808-4400-b954-e13985b00908-secret-telemeter-client\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.132114 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.132100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be1bfa15-8808-4400-b954-e13985b00908-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.138128 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.138094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbfr9\" (UniqueName: \"kubernetes.io/projected/be1bfa15-8808-4400-b954-e13985b00908-kube-api-access-wbfr9\") pod \"telemeter-client-68f4d884dc-fxk4v\" (UID: \"be1bfa15-8808-4400-b954-e13985b00908\") " pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.285337 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.285297    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"
Sep 16 18:48:23.564810 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.564784    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68f4d884dc-fxk4v"]
Sep 16 18:48:23.566629 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:48:23.566601    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1bfa15_8808_4400_b954_e13985b00908.slice/crio-d7b5c6b775e6bff5999222d5e1d7c62d80787ad9ff84bc75b4a97982b0913a13 WatchSource:0}: Error finding container d7b5c6b775e6bff5999222d5e1d7c62d80787ad9ff84bc75b4a97982b0913a13: Status 404 returned error can't find the container with id d7b5c6b775e6bff5999222d5e1d7c62d80787ad9ff84bc75b4a97982b0913a13
Sep 16 18:48:23.581746 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:23.581714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v" event={"ID":"be1bfa15-8808-4400-b954-e13985b00908","Type":"ContainerStarted","Data":"d7b5c6b775e6bff5999222d5e1d7c62d80787ad9ff84bc75b4a97982b0913a13"}
Sep 16 18:48:25.595234 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.595103    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v" event={"ID":"be1bfa15-8808-4400-b954-e13985b00908","Type":"ContainerStarted","Data":"8f1b71ca53c835d146f82c48c7cb9b844d9b90478618f09d03edfca4a626110c"}
Sep 16 18:48:25.595234 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.595182    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v" event={"ID":"be1bfa15-8808-4400-b954-e13985b00908","Type":"ContainerStarted","Data":"048661b7d09402764efe37c1ffd5e56e912c72e48de9ba041202a7453158fa5c"}
Sep 16 18:48:25.595234 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.595211    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v" event={"ID":"be1bfa15-8808-4400-b954-e13985b00908","Type":"ContainerStarted","Data":"74336c9bafbf15bd829fae88e9c68907d47bb4fa39a987a54439e3b1987586fe"}
Sep 16 18:48:25.624054 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.623990    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-68f4d884dc-fxk4v" podStartSLOduration=1.923590233 podStartE2EDuration="3.623970406s" totalImagesPullingTime="1.700380173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:25.622992116 +0000 UTC m=+163.453600673" watchObservedRunningTime="2026-09-16 18:48:25.623970406 +0000 UTC m=+163.454578950"
Sep 16 18:48:25.715732 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.715697    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-655d779c9d-g6tsh"]
Sep 16 18:48:25.718557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.718529    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.731349 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.731321    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-655d779c9d-g6tsh"]
Sep 16 18:48:25.772255 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.772220    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:25.833426 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.833395    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:25.857712 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.857631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-oauth-config\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.857712 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.857708    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-service-ca\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.857881 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.857732    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-serving-cert\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.857881 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.857762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-trusted-ca-bundle\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.857881 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.857816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knl9l\" (UniqueName: \"kubernetes.io/projected/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-kube-api-access-knl9l\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.857881 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.857865    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-config\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.858019 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.857889    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-oauth-serving-cert\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.958539 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.958501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knl9l\" (UniqueName: \"kubernetes.io/projected/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-kube-api-access-knl9l\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.958539 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.958549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-config\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.958809 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.958570    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-oauth-serving-cert\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.958809 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.958615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-oauth-config\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.958809 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.958711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-service-ca\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.958809 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.958741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-serving-cert\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.958809 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.958782    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-trusted-ca-bundle\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.959467 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.959424    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-config\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.959467 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.959451    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-oauth-serving-cert\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.959637 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.959515    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-service-ca\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.959682 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.959673    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-trusted-ca-bundle\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.961438 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.961406    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-oauth-config\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.961620 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.961564    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-serving-cert\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:25.967157 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:25.967136    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knl9l\" (UniqueName: \"kubernetes.io/projected/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-kube-api-access-knl9l\") pod \"console-655d779c9d-g6tsh\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") " pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:26.027871 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:26.027819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:26.309705 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:26.309492    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-655d779c9d-g6tsh"]
Sep 16 18:48:26.312329 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:48:26.312295    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa9bc2a_6f89_44ad_ae8f_ec3d6e50adb9.slice/crio-24d24b48016d8c5393ccf8887ff937306d56c6538716d8f1dd2cda3a92db572f WatchSource:0}: Error finding container 24d24b48016d8c5393ccf8887ff937306d56c6538716d8f1dd2cda3a92db572f: Status 404 returned error can't find the container with id 24d24b48016d8c5393ccf8887ff937306d56c6538716d8f1dd2cda3a92db572f
Sep 16 18:48:26.601397 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:26.601303    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655d779c9d-g6tsh" event={"ID":"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9","Type":"ContainerStarted","Data":"980ef12bb81ad3d9b5988f16fddbd26262c68576117f42149a080c13303ba76f"}
Sep 16 18:48:26.601397 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:26.601369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655d779c9d-g6tsh" event={"ID":"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9","Type":"ContainerStarted","Data":"24d24b48016d8c5393ccf8887ff937306d56c6538716d8f1dd2cda3a92db572f"}
Sep 16 18:48:26.623796 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:26.623735    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-655d779c9d-g6tsh" podStartSLOduration=1.623717844 podStartE2EDuration="1.623717844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:26.622443326 +0000 UTC m=+164.453051871" watchObservedRunningTime="2026-09-16 18:48:26.623717844 +0000 UTC m=+164.454326388"
Sep 16 18:48:36.028346 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:36.028259    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:36.028346 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:36.028325    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:36.033142 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:36.033110    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:36.649050 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:36.649021    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:48:36.695387 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:36.695349    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-df5dbc8fc-h5l92"]
Sep 16 18:48:36.799125 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:36.799086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:36.900293 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:36.900194    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:36.956995 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:48:36.956962    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:01.716536 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:01.716462    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-df5dbc8fc-h5l92" podUID="78ef053c-664c-4555-98be-fbf381dd65e2" containerName="console" containerID="cri-o://dc55106ba80ec3e616d65380ee1f8cebafc81d8736beb63862fc16884f7b67c7" gracePeriod=15
Sep 16 18:49:01.953374 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:01.953342    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:49:02.088345 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.088307    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-console-config\") pod \"78ef053c-664c-4555-98be-fbf381dd65e2\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") "
Sep 16 18:49:02.088345 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.088345    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-service-ca\") pod \"78ef053c-664c-4555-98be-fbf381dd65e2\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") "
Sep 16 18:49:02.088569 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.088368    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-trusted-ca-bundle\") pod \"78ef053c-664c-4555-98be-fbf381dd65e2\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") "
Sep 16 18:49:02.088569 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.088409    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78ef053c-664c-4555-98be-fbf381dd65e2-console-oauth-config\") pod \"78ef053c-664c-4555-98be-fbf381dd65e2\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") "
Sep 16 18:49:02.088791 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.088653    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6w2n\" (UniqueName: \"kubernetes.io/projected/78ef053c-664c-4555-98be-fbf381dd65e2-kube-api-access-c6w2n\") pod \"78ef053c-664c-4555-98be-fbf381dd65e2\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") "
Sep 16 18:49:02.088791 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.088731    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78ef053c-664c-4555-98be-fbf381dd65e2-console-serving-cert\") pod \"78ef053c-664c-4555-98be-fbf381dd65e2\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") "
Sep 16 18:49:02.088791 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.088764    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-oauth-serving-cert\") pod \"78ef053c-664c-4555-98be-fbf381dd65e2\" (UID: \"78ef053c-664c-4555-98be-fbf381dd65e2\") "
Sep 16 18:49:02.088928 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.088844    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-service-ca" (OuterVolumeSpecName: "service-ca") pod "78ef053c-664c-4555-98be-fbf381dd65e2" (UID: "78ef053c-664c-4555-98be-fbf381dd65e2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:02.088928 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.088852    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-console-config" (OuterVolumeSpecName: "console-config") pod "78ef053c-664c-4555-98be-fbf381dd65e2" (UID: "78ef053c-664c-4555-98be-fbf381dd65e2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:02.088928 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.088862    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "78ef053c-664c-4555-98be-fbf381dd65e2" (UID: "78ef053c-664c-4555-98be-fbf381dd65e2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:02.089149 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.089131    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-console-config\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:49:02.089149 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.089150    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-service-ca\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:49:02.089245 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.089158    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-trusted-ca-bundle\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:49:02.089245 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.089170    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "78ef053c-664c-4555-98be-fbf381dd65e2" (UID: "78ef053c-664c-4555-98be-fbf381dd65e2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:02.090873 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.090854    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78ef053c-664c-4555-98be-fbf381dd65e2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "78ef053c-664c-4555-98be-fbf381dd65e2" (UID: "78ef053c-664c-4555-98be-fbf381dd65e2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:02.090949 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.090922    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ef053c-664c-4555-98be-fbf381dd65e2-kube-api-access-c6w2n" (OuterVolumeSpecName: "kube-api-access-c6w2n") pod "78ef053c-664c-4555-98be-fbf381dd65e2" (UID: "78ef053c-664c-4555-98be-fbf381dd65e2"). InnerVolumeSpecName "kube-api-access-c6w2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:02.090986 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.090974    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78ef053c-664c-4555-98be-fbf381dd65e2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "78ef053c-664c-4555-98be-fbf381dd65e2" (UID: "78ef053c-664c-4555-98be-fbf381dd65e2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:02.190331 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.190276    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78ef053c-664c-4555-98be-fbf381dd65e2-console-serving-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:49:02.190331 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.190320    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78ef053c-664c-4555-98be-fbf381dd65e2-oauth-serving-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:49:02.190331 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.190329    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78ef053c-664c-4555-98be-fbf381dd65e2-console-oauth-config\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:49:02.190331 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.190338    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6w2n\" (UniqueName: \"kubernetes.io/projected/78ef053c-664c-4555-98be-fbf381dd65e2-kube-api-access-c6w2n\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:49:02.748850 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.748811    2582 generic.go:356] "Generic (PLEG): container finished" podID="78ef053c-664c-4555-98be-fbf381dd65e2" containerID="dc55106ba80ec3e616d65380ee1f8cebafc81d8736beb63862fc16884f7b67c7" exitCode=0
Sep 16 18:49:02.749265 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.748860    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df5dbc8fc-h5l92" event={"ID":"78ef053c-664c-4555-98be-fbf381dd65e2","Type":"ContainerDied","Data":"dc55106ba80ec3e616d65380ee1f8cebafc81d8736beb63862fc16884f7b67c7"}
Sep 16 18:49:02.749265 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.748867    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df5dbc8fc-h5l92"
Sep 16 18:49:02.749265 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.748907    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df5dbc8fc-h5l92" event={"ID":"78ef053c-664c-4555-98be-fbf381dd65e2","Type":"ContainerDied","Data":"1f1ef029f640b2dba3a50ac809d5fffc5d790dc4ac32de4b48479d2db23b5a9d"}
Sep 16 18:49:02.749265 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.748928    2582 scope.go:117] "RemoveContainer" containerID="dc55106ba80ec3e616d65380ee1f8cebafc81d8736beb63862fc16884f7b67c7"
Sep 16 18:49:02.757830 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.757805    2582 scope.go:117] "RemoveContainer" containerID="dc55106ba80ec3e616d65380ee1f8cebafc81d8736beb63862fc16884f7b67c7"
Sep 16 18:49:02.758155 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:49:02.758135    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc55106ba80ec3e616d65380ee1f8cebafc81d8736beb63862fc16884f7b67c7\": container with ID starting with dc55106ba80ec3e616d65380ee1f8cebafc81d8736beb63862fc16884f7b67c7 not found: ID does not exist" containerID="dc55106ba80ec3e616d65380ee1f8cebafc81d8736beb63862fc16884f7b67c7"
Sep 16 18:49:02.758235 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.758163    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc55106ba80ec3e616d65380ee1f8cebafc81d8736beb63862fc16884f7b67c7"} err="failed to get container status \"dc55106ba80ec3e616d65380ee1f8cebafc81d8736beb63862fc16884f7b67c7\": rpc error: code = NotFound desc = could not find container \"dc55106ba80ec3e616d65380ee1f8cebafc81d8736beb63862fc16884f7b67c7\": container with ID starting with dc55106ba80ec3e616d65380ee1f8cebafc81d8736beb63862fc16884f7b67c7 not found: ID does not exist"
Sep 16 18:49:02.766851 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.766824    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-df5dbc8fc-h5l92"]
Sep 16 18:49:02.770653 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:02.770628    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-df5dbc8fc-h5l92"]
Sep 16 18:49:04.704122 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:04.704076    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78ef053c-664c-4555-98be-fbf381dd65e2" path="/var/lib/kubelet/pods/78ef053c-664c-4555-98be-fbf381dd65e2/volumes"
Sep 16 18:49:24.355549 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:24.355492    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:24.411460 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:24.411426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:33.097644 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:33.097602    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:33.162626 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:33.162590    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:37.660267 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.660228    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8675b48b47-v4n6f"]
Sep 16 18:49:37.660729 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.660564    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78ef053c-664c-4555-98be-fbf381dd65e2" containerName="console"
Sep 16 18:49:37.660729 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.660590    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="78ef053c-664c-4555-98be-fbf381dd65e2" containerName="console"
Sep 16 18:49:37.660729 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.660675    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="78ef053c-664c-4555-98be-fbf381dd65e2" containerName="console"
Sep 16 18:49:37.662833 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.662814    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.677042 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.677014    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8675b48b47-v4n6f"]
Sep 16 18:49:37.718228 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.718189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-serving-cert\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.718228 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.718237    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-oauth-serving-cert\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.718493 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.718318    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-trusted-ca-bundle\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.718493 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.718384    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-service-ca\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.718493 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.718420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-config\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.718493 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.718470    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-oauth-config\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.718714 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.718500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfmsm\" (UniqueName: \"kubernetes.io/projected/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-kube-api-access-mfmsm\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.746541 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.746502    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:37.803412 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.803369    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:37.819569 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.819535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-oauth-serving-cert\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.819720 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.819606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-trusted-ca-bundle\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.819720 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.819632    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-service-ca\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.819720 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.819656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-config\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.819720 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.819680    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-oauth-config\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.819720 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.819707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfmsm\" (UniqueName: \"kubernetes.io/projected/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-kube-api-access-mfmsm\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.820056 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.819802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-serving-cert\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.820507 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.820480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-oauth-serving-cert\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.820665 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.820506    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-config\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.820665 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.820492    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-service-ca\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.820767 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.820669    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-trusted-ca-bundle\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.822612 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.822567    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-oauth-config\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.822809 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.822789    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-serving-cert\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.832297 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.832267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfmsm\" (UniqueName: \"kubernetes.io/projected/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-kube-api-access-mfmsm\") pod \"console-8675b48b47-v4n6f\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") " pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:37.873192 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.873140    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:37.973506 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:37.973387    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:38.252982 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:38.252889    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8675b48b47-v4n6f"]
Sep 16 18:49:38.256969 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:49:38.256933    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3412f7e2_2655_4ddb_9f46_bbac0b5edeb4.slice/crio-e059e768a692b3fb5cd173ceba0384a2c3d760e692e76a2a8ddc7435b87b9a7c WatchSource:0}: Error finding container e059e768a692b3fb5cd173ceba0384a2c3d760e692e76a2a8ddc7435b87b9a7c: Status 404 returned error can't find the container with id e059e768a692b3fb5cd173ceba0384a2c3d760e692e76a2a8ddc7435b87b9a7c
Sep 16 18:49:38.461718 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:38.461682    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:38.523902 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:38.523866    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:38.900942 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:38.900846    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8675b48b47-v4n6f" event={"ID":"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4","Type":"ContainerStarted","Data":"27a85cc365b33dcf9111cb3027a77e22391ffc609c23486df023fd1ee007e8c0"}
Sep 16 18:49:38.900942 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:38.900913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8675b48b47-v4n6f" event={"ID":"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4","Type":"ContainerStarted","Data":"e059e768a692b3fb5cd173ceba0384a2c3d760e692e76a2a8ddc7435b87b9a7c"}
Sep 16 18:49:38.941224 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:38.941159    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8675b48b47-v4n6f" podStartSLOduration=1.941135711 podStartE2EDuration="1.941135711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:38.939956577 +0000 UTC m=+236.770565122" watchObservedRunningTime="2026-09-16 18:49:38.941135711 +0000 UTC m=+236.771744312"
Sep 16 18:49:44.994068 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:44.994021    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:45.052837 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:45.052798    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:47.713393 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:47.713355    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:47.814214 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:47.814183    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:47.973664 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:47.973527    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:47.973664 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:47.973635    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:47.978841 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:47.978817    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:48.951496 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:48.951464    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:49:49.000999 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:49.000953    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-655d779c9d-g6tsh"]
Sep 16 18:49:49.114065 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:49.114027    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:49.216823 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:49.216686    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:49.316886 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:49:49.316853    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:14.024992 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.024930    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-655d779c9d-g6tsh" podUID="1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9" containerName="console" containerID="cri-o://980ef12bb81ad3d9b5988f16fddbd26262c68576117f42149a080c13303ba76f" gracePeriod=15
Sep 16 18:50:14.257914 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.257890    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:50:14.339867 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.339779    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-service-ca\") pod \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") "
Sep 16 18:50:14.339867 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.339827    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-config\") pod \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") "
Sep 16 18:50:14.339867 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.339843    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-oauth-serving-cert\") pod \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") "
Sep 16 18:50:14.339867 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.339861    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-serving-cert\") pod \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") "
Sep 16 18:50:14.340209 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.339880    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-trusted-ca-bundle\") pod \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") "
Sep 16 18:50:14.340209 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.339976    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-oauth-config\") pod \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") "
Sep 16 18:50:14.340209 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.340000    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knl9l\" (UniqueName: \"kubernetes.io/projected/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-kube-api-access-knl9l\") pod \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\" (UID: \"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9\") "
Sep 16 18:50:14.340383 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.340341    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-config" (OuterVolumeSpecName: "console-config") pod "1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9" (UID: "1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:14.340383 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.340361    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9" (UID: "1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:14.340502 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.340455    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9" (UID: "1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:14.340556 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.340504    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-service-ca" (OuterVolumeSpecName: "service-ca") pod "1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9" (UID: "1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:14.342335 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.342312    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9" (UID: "1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:14.342441 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.342333    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9" (UID: "1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:14.342441 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.342403    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-kube-api-access-knl9l" (OuterVolumeSpecName: "kube-api-access-knl9l") pod "1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9" (UID: "1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9"). InnerVolumeSpecName "kube-api-access-knl9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:14.441177 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.441137    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-oauth-config\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:50:14.441177 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.441165    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knl9l\" (UniqueName: \"kubernetes.io/projected/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-kube-api-access-knl9l\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:50:14.441177 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.441176    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-service-ca\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:50:14.441177 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.441185    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-config\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:50:14.441431 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.441193    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-oauth-serving-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:50:14.441431 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.441201    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-console-serving-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:50:14.441431 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.441210    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9-trusted-ca-bundle\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:50:14.642846 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.642755    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:14.742550 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.742514    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:14.796513 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:14.796482    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:15.051119 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:15.051077    2582 generic.go:356] "Generic (PLEG): container finished" podID="1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9" containerID="980ef12bb81ad3d9b5988f16fddbd26262c68576117f42149a080c13303ba76f" exitCode=0
Sep 16 18:50:15.051119 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:15.051128    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655d779c9d-g6tsh"
Sep 16 18:50:15.051650 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:15.051157    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655d779c9d-g6tsh" event={"ID":"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9","Type":"ContainerDied","Data":"980ef12bb81ad3d9b5988f16fddbd26262c68576117f42149a080c13303ba76f"}
Sep 16 18:50:15.051650 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:15.051209    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655d779c9d-g6tsh" event={"ID":"1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9","Type":"ContainerDied","Data":"24d24b48016d8c5393ccf8887ff937306d56c6538716d8f1dd2cda3a92db572f"}
Sep 16 18:50:15.051650 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:15.051231    2582 scope.go:117] "RemoveContainer" containerID="980ef12bb81ad3d9b5988f16fddbd26262c68576117f42149a080c13303ba76f"
Sep 16 18:50:15.060623 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:15.060597    2582 scope.go:117] "RemoveContainer" containerID="980ef12bb81ad3d9b5988f16fddbd26262c68576117f42149a080c13303ba76f"
Sep 16 18:50:15.060952 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:50:15.060933    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980ef12bb81ad3d9b5988f16fddbd26262c68576117f42149a080c13303ba76f\": container with ID starting with 980ef12bb81ad3d9b5988f16fddbd26262c68576117f42149a080c13303ba76f not found: ID does not exist" containerID="980ef12bb81ad3d9b5988f16fddbd26262c68576117f42149a080c13303ba76f"
Sep 16 18:50:15.061015 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:15.060961    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980ef12bb81ad3d9b5988f16fddbd26262c68576117f42149a080c13303ba76f"} err="failed to get container status \"980ef12bb81ad3d9b5988f16fddbd26262c68576117f42149a080c13303ba76f\": rpc error: code = NotFound desc = could not find container \"980ef12bb81ad3d9b5988f16fddbd26262c68576117f42149a080c13303ba76f\": container with ID starting with 980ef12bb81ad3d9b5988f16fddbd26262c68576117f42149a080c13303ba76f not found: ID does not exist"
Sep 16 18:50:15.072299 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:15.072261    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-655d779c9d-g6tsh"]
Sep 16 18:50:15.078899 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:15.078874    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-655d779c9d-g6tsh"]
Sep 16 18:50:16.703357 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:16.703323    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9" path="/var/lib/kubelet/pods/1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9/volumes"
Sep 16 18:50:20.670917 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:20.670886    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:42.015226 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:42.015182    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:42.590928 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:42.590894    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:50:43.290793 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.290753    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:43.307187 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.307153    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r"]
Sep 16 18:50:43.307493 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.307481    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9" containerName="console"
Sep 16 18:50:43.307538 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.307496    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9" containerName="console"
Sep 16 18:50:43.307595 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.307557    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aa9bc2a-6f89-44ad-ae8f-ec3d6e50adb9" containerName="console"
Sep 16 18:50:43.309546 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.309530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r"
Sep 16 18:50:43.312555 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.312529    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:50:43.312707 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.312544    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:50:43.312707 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.312631    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-9k9gk\""
Sep 16 18:50:43.312707 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.312635    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:50:43.328955 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.328925    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r"]
Sep 16 18:50:43.356835 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.356795    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhv2k\" (UniqueName: \"kubernetes.io/projected/b66738ff-35c2-4c3d-8202-3d50cce8dac6-kube-api-access-xhv2k\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9mq2r\" (UID: \"b66738ff-35c2-4c3d-8202-3d50cce8dac6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r"
Sep 16 18:50:43.357036 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.356847    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b66738ff-35c2-4c3d-8202-3d50cce8dac6-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9mq2r\" (UID: \"b66738ff-35c2-4c3d-8202-3d50cce8dac6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r"
Sep 16 18:50:43.457439 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.457394    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b66738ff-35c2-4c3d-8202-3d50cce8dac6-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9mq2r\" (UID: \"b66738ff-35c2-4c3d-8202-3d50cce8dac6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r"
Sep 16 18:50:43.457678 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.457550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhv2k\" (UniqueName: \"kubernetes.io/projected/b66738ff-35c2-4c3d-8202-3d50cce8dac6-kube-api-access-xhv2k\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9mq2r\" (UID: \"b66738ff-35c2-4c3d-8202-3d50cce8dac6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r"
Sep 16 18:50:43.460210 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.460185    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b66738ff-35c2-4c3d-8202-3d50cce8dac6-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9mq2r\" (UID: \"b66738ff-35c2-4c3d-8202-3d50cce8dac6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r"
Sep 16 18:50:43.476665 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.476634    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhv2k\" (UniqueName: \"kubernetes.io/projected/b66738ff-35c2-4c3d-8202-3d50cce8dac6-kube-api-access-xhv2k\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-9mq2r\" (UID: \"b66738ff-35c2-4c3d-8202-3d50cce8dac6\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r"
Sep 16 18:50:43.619808 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.619704    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r"
Sep 16 18:50:43.900273 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.900248    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r"]
Sep 16 18:50:43.901949 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:50:43.901922    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66738ff_35c2_4c3d_8202_3d50cce8dac6.slice/crio-ae139f0065c1cdcf1b3f6c45f978d971594b3a9d872919e7af93a3dd756a94fe WatchSource:0}: Error finding container ae139f0065c1cdcf1b3f6c45f978d971594b3a9d872919e7af93a3dd756a94fe: Status 404 returned error can't find the container with id ae139f0065c1cdcf1b3f6c45f978d971594b3a9d872919e7af93a3dd756a94fe
Sep 16 18:50:43.903914 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:43.903892    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:50:44.171750 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:44.171656    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r" event={"ID":"b66738ff-35c2-4c3d-8202-3d50cce8dac6","Type":"ContainerStarted","Data":"ae139f0065c1cdcf1b3f6c45f978d971594b3a9d872919e7af93a3dd756a94fe"}
Sep 16 18:50:47.625726 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:47.625668    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:47.907184 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:47.907091    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:48.194055 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:48.193964    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r" event={"ID":"b66738ff-35c2-4c3d-8202-3d50cce8dac6","Type":"ContainerStarted","Data":"ae0c220ed3a2d7e4d20e34c78889b04ed5e245e5c3638fb2a8e9bd9299f5c708"}
Sep 16 18:50:48.194196 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:48.194081    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r"
Sep 16 18:50:48.195853 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:48.195831    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:48.217242 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:48.217175    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r" podStartSLOduration=1.924037266 podStartE2EDuration="5.217150973s" totalImagesPullingTime="3.293113707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:48.215735667 +0000 UTC m=+306.046344212" watchObservedRunningTime="2026-09-16 18:50:48.217150973 +0000 UTC m=+306.047759521"
Sep 16 18:50:48.358144 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:48.358105    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:49.037557 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:49.037528    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:49.125319 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:49.125287    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:49.362047 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:49.361969    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:49.560057 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:49.560022    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:49.738510 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:49.738444    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:49.807233 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:50:49.807210    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:09.200207 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:51:09.200171    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-9mq2r"
Sep 16 18:51:40.530992 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:51:40.530888    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:40.622491 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:51:40.622452    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:51.679856 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:51:51.679813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:52.406709 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:51:52.406653    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:54.345428 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:51:54.345388    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:54.446100 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:51:54.446065    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:55.308057 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:51:55.308018    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:55.408918 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:51:55.408877    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:57.513406 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:51:57.513370    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:57.607968 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:51:57.607930    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:59.686905 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:51:59.686869    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:00.080820 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.080786    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:00.181746 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.181708    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:00.616084 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.616034    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qmn5m"]
Sep 16 18:52:00.618801 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.618783    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmn5m"
Sep 16 18:52:00.624472 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.624440    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 18:52:00.624688 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.624647    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 18:52:00.626076 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.626055    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-l8xlf\""
Sep 16 18:52:00.631499 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.631473    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qmn5m"]
Sep 16 18:52:00.731440 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.731382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47869257-08c9-46e2-85fe-bd15b3767c91-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qmn5m\" (UID: \"47869257-08c9-46e2-85fe-bd15b3767c91\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmn5m"
Sep 16 18:52:00.732022 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.731483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mndz\" (UniqueName: \"kubernetes.io/projected/47869257-08c9-46e2-85fe-bd15b3767c91-kube-api-access-6mndz\") pod \"cert-manager-cainjector-559cdd8c87-qmn5m\" (UID: \"47869257-08c9-46e2-85fe-bd15b3767c91\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmn5m"
Sep 16 18:52:00.832819 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.832778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mndz\" (UniqueName: \"kubernetes.io/projected/47869257-08c9-46e2-85fe-bd15b3767c91-kube-api-access-6mndz\") pod \"cert-manager-cainjector-559cdd8c87-qmn5m\" (UID: \"47869257-08c9-46e2-85fe-bd15b3767c91\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmn5m"
Sep 16 18:52:00.833002 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.832861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47869257-08c9-46e2-85fe-bd15b3767c91-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qmn5m\" (UID: \"47869257-08c9-46e2-85fe-bd15b3767c91\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmn5m"
Sep 16 18:52:00.843104 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.843073    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47869257-08c9-46e2-85fe-bd15b3767c91-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qmn5m\" (UID: \"47869257-08c9-46e2-85fe-bd15b3767c91\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmn5m"
Sep 16 18:52:00.843450 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.843424    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mndz\" (UniqueName: \"kubernetes.io/projected/47869257-08c9-46e2-85fe-bd15b3767c91-kube-api-access-6mndz\") pod \"cert-manager-cainjector-559cdd8c87-qmn5m\" (UID: \"47869257-08c9-46e2-85fe-bd15b3767c91\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmn5m"
Sep 16 18:52:00.929263 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:00.929153    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmn5m"
Sep 16 18:52:01.204917 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:01.204886    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qmn5m"]
Sep 16 18:52:01.208100 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:52:01.208068    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47869257_08c9_46e2_85fe_bd15b3767c91.slice/crio-3fd57670062956f508fe70d796a3d15657e674334cf49668ed15c17510bca183 WatchSource:0}: Error finding container 3fd57670062956f508fe70d796a3d15657e674334cf49668ed15c17510bca183: Status 404 returned error can't find the container with id 3fd57670062956f508fe70d796a3d15657e674334cf49668ed15c17510bca183
Sep 16 18:52:01.513090 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:01.512980    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmn5m" event={"ID":"47869257-08c9-46e2-85fe-bd15b3767c91","Type":"ContainerStarted","Data":"3fd57670062956f508fe70d796a3d15657e674334cf49668ed15c17510bca183"}
Sep 16 18:52:02.025753 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:02.025715    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:02.133493 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:02.133437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:02.231548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:02.231499    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:04.531906 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:04.531868    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmn5m" event={"ID":"47869257-08c9-46e2-85fe-bd15b3767c91","Type":"ContainerStarted","Data":"f3585ac0ce0ed62dc6e566495e0e4eca3a7b8cfd37c3d050cc8cb8beaa430c72"}
Sep 16 18:52:04.547968 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:04.547919    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmn5m" podStartSLOduration=1.361778887 podStartE2EDuration="4.547902282s" totalImagesPullingTime="3.186123395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:04.547226236 +0000 UTC m=+382.377834782" watchObservedRunningTime="2026-09-16 18:52:04.547902282 +0000 UTC m=+382.378510826"
Sep 16 18:52:05.284683 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:05.284645    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:05.385587 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:05.385553    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:07.697119 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:07.697081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:13.889755 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:13.889711    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:13.984913 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:13.984867    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:15.909150 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:15.909105    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wk7f5"]
Sep 16 18:52:15.912129 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:15.912102    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wk7f5"
Sep 16 18:52:15.915269 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:15.915244    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 18:52:15.916559 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:15.916539    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zlkdb\""
Sep 16 18:52:15.916559 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:15.916546    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:52:15.921035 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:15.921011    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wk7f5"]
Sep 16 18:52:15.958285 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:15.958246    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7221fc7-908b-43bb-843f-a9ca32915b4b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wk7f5\" (UID: \"f7221fc7-908b-43bb-843f-a9ca32915b4b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wk7f5"
Sep 16 18:52:15.958472 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:15.958306    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52ldl\" (UniqueName: \"kubernetes.io/projected/f7221fc7-908b-43bb-843f-a9ca32915b4b-kube-api-access-52ldl\") pod \"openshift-lws-operator-fd8ccff4c-wk7f5\" (UID: \"f7221fc7-908b-43bb-843f-a9ca32915b4b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wk7f5"
Sep 16 18:52:16.058923 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:16.058875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7221fc7-908b-43bb-843f-a9ca32915b4b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wk7f5\" (UID: \"f7221fc7-908b-43bb-843f-a9ca32915b4b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wk7f5"
Sep 16 18:52:16.059168 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:16.058969    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52ldl\" (UniqueName: \"kubernetes.io/projected/f7221fc7-908b-43bb-843f-a9ca32915b4b-kube-api-access-52ldl\") pod \"openshift-lws-operator-fd8ccff4c-wk7f5\" (UID: \"f7221fc7-908b-43bb-843f-a9ca32915b4b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wk7f5"
Sep 16 18:52:16.059446 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:16.059419    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7221fc7-908b-43bb-843f-a9ca32915b4b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wk7f5\" (UID: \"f7221fc7-908b-43bb-843f-a9ca32915b4b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wk7f5"
Sep 16 18:52:16.068462 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:16.068427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52ldl\" (UniqueName: \"kubernetes.io/projected/f7221fc7-908b-43bb-843f-a9ca32915b4b-kube-api-access-52ldl\") pod \"openshift-lws-operator-fd8ccff4c-wk7f5\" (UID: \"f7221fc7-908b-43bb-843f-a9ca32915b4b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wk7f5"
Sep 16 18:52:16.223518 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:16.223414    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wk7f5"
Sep 16 18:52:16.500223 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:16.500127    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wk7f5"]
Sep 16 18:52:16.503524 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:52:16.503490    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7221fc7_908b_43bb_843f_a9ca32915b4b.slice/crio-b53a5842c51f0e7ed4e7154be1e8290b83069a30b2aa4837ebb772d9d9c9f345 WatchSource:0}: Error finding container b53a5842c51f0e7ed4e7154be1e8290b83069a30b2aa4837ebb772d9d9c9f345: Status 404 returned error can't find the container with id b53a5842c51f0e7ed4e7154be1e8290b83069a30b2aa4837ebb772d9d9c9f345
Sep 16 18:52:16.588028 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:16.587988    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wk7f5" event={"ID":"f7221fc7-908b-43bb-843f-a9ca32915b4b","Type":"ContainerStarted","Data":"b53a5842c51f0e7ed4e7154be1e8290b83069a30b2aa4837ebb772d9d9c9f345"}
Sep 16 18:52:32.664649 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:32.664617    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wk7f5" event={"ID":"f7221fc7-908b-43bb-843f-a9ca32915b4b","Type":"ContainerStarted","Data":"67ac6ab0eb277f0e132760fa77db4caf67beded1c1fd060a71fde9ee94c09723"}
Sep 16 18:52:32.688111 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:32.688062    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wk7f5" podStartSLOduration=1.9790179879999998 podStartE2EDuration="17.688046959s" totalImagesPullingTime="15.709028971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:32.686174302 +0000 UTC m=+410.516782850" watchObservedRunningTime="2026-09-16 18:52:32.688046959 +0000 UTC m=+410.518655503"
Sep 16 18:52:36.475842 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:36.475812    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:40.768366 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:40.768336    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:40.855181 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:40.855149    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"]
Sep 16 18:52:40.859908 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:40.859892    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:40.864265 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:40.864241    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 18:52:40.864389 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:40.864263    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 18:52:40.864389 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:40.864290    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jqqt9\""
Sep 16 18:52:40.864389 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:40.864246    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 18:52:40.872071 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:40.872051    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:40.875312 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:40.875285    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"]
Sep 16 18:52:40.968847 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:40.968819    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa242294-121a-44b3-84d6-ffbaae0b1472-cert\") pod \"lws-controller-manager-6c6db7499f-7b9dh\" (UID: \"fa242294-121a-44b3-84d6-ffbaae0b1472\") " pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:40.968964 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:40.968860    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6xh2\" (UniqueName: \"kubernetes.io/projected/fa242294-121a-44b3-84d6-ffbaae0b1472-kube-api-access-d6xh2\") pod \"lws-controller-manager-6c6db7499f-7b9dh\" (UID: \"fa242294-121a-44b3-84d6-ffbaae0b1472\") " pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:40.968964 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:40.968891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fa242294-121a-44b3-84d6-ffbaae0b1472-manager-config\") pod \"lws-controller-manager-6c6db7499f-7b9dh\" (UID: \"fa242294-121a-44b3-84d6-ffbaae0b1472\") " pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:40.968964 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:40.968929    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa242294-121a-44b3-84d6-ffbaae0b1472-metrics-cert\") pod \"lws-controller-manager-6c6db7499f-7b9dh\" (UID: \"fa242294-121a-44b3-84d6-ffbaae0b1472\") " pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:41.040235 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:41.040174    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:41.070251 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:41.070228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa242294-121a-44b3-84d6-ffbaae0b1472-cert\") pod \"lws-controller-manager-6c6db7499f-7b9dh\" (UID: \"fa242294-121a-44b3-84d6-ffbaae0b1472\") " pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:41.070351 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:41.070253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6xh2\" (UniqueName: \"kubernetes.io/projected/fa242294-121a-44b3-84d6-ffbaae0b1472-kube-api-access-d6xh2\") pod \"lws-controller-manager-6c6db7499f-7b9dh\" (UID: \"fa242294-121a-44b3-84d6-ffbaae0b1472\") " pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:41.070351 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:41.070275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fa242294-121a-44b3-84d6-ffbaae0b1472-manager-config\") pod \"lws-controller-manager-6c6db7499f-7b9dh\" (UID: \"fa242294-121a-44b3-84d6-ffbaae0b1472\") " pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:41.070351 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:41.070295    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa242294-121a-44b3-84d6-ffbaae0b1472-metrics-cert\") pod \"lws-controller-manager-6c6db7499f-7b9dh\" (UID: \"fa242294-121a-44b3-84d6-ffbaae0b1472\") " pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:41.071435 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:41.071406    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fa242294-121a-44b3-84d6-ffbaae0b1472-manager-config\") pod \"lws-controller-manager-6c6db7499f-7b9dh\" (UID: \"fa242294-121a-44b3-84d6-ffbaae0b1472\") " pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:41.078491 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:41.078469    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa242294-121a-44b3-84d6-ffbaae0b1472-metrics-cert\") pod \"lws-controller-manager-6c6db7499f-7b9dh\" (UID: \"fa242294-121a-44b3-84d6-ffbaae0b1472\") " pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:41.078652 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:41.078533    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa242294-121a-44b3-84d6-ffbaae0b1472-cert\") pod \"lws-controller-manager-6c6db7499f-7b9dh\" (UID: \"fa242294-121a-44b3-84d6-ffbaae0b1472\") " pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:41.088916 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:41.088893    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6xh2\" (UniqueName: \"kubernetes.io/projected/fa242294-121a-44b3-84d6-ffbaae0b1472-kube-api-access-d6xh2\") pod \"lws-controller-manager-6c6db7499f-7b9dh\" (UID: \"fa242294-121a-44b3-84d6-ffbaae0b1472\") " pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:41.142137 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:41.142111    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:41.169114 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:41.169093    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:41.440031 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:41.439959    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"]
Sep 16 18:52:41.442817 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:52:41.442792    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa242294_121a_44b3_84d6_ffbaae0b1472.slice/crio-d6072fe6ef39bc5529f1b11ac9c358519afe58b83dc37a447f66c73695242736 WatchSource:0}: Error finding container d6072fe6ef39bc5529f1b11ac9c358519afe58b83dc37a447f66c73695242736: Status 404 returned error can't find the container with id d6072fe6ef39bc5529f1b11ac9c358519afe58b83dc37a447f66c73695242736
Sep 16 18:52:41.705680 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:41.705599    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh" event={"ID":"fa242294-121a-44b3-84d6-ffbaae0b1472","Type":"ContainerStarted","Data":"d6072fe6ef39bc5529f1b11ac9c358519afe58b83dc37a447f66c73695242736"}
Sep 16 18:52:42.198051 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:42.198020    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:43.715265 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:43.715236    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh" event={"ID":"fa242294-121a-44b3-84d6-ffbaae0b1472","Type":"ContainerStarted","Data":"c5080b3a74e65f29211c634a5252fb772e980a685b9da244aac782e942ff2db2"}
Sep 16 18:52:43.715651 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:43.715359    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:43.733819 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:43.733771    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh" podStartSLOduration=2.346296041 podStartE2EDuration="3.73375571s" totalImagesPullingTime="1.387459669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:43.732384895 +0000 UTC m=+421.562993441" watchObservedRunningTime="2026-09-16 18:52:43.73375571 +0000 UTC m=+421.564364255"
Sep 16 18:52:49.379698 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:49.379661    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:53.501197 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:53.501156    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:53.584468 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:53.584416    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:54.721712 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:54.721675    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c6db7499f-7b9dh"
Sep 16 18:52:56.826000 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:56.825955    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:59.014865 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:59.014829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:59.115949 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:52:59.115911    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:17.952081 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:17.952051    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:18.373122 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:18.373089    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:18.474636 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:18.474596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:20.122921 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:20.122881    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:20.337863 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:20.337838    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:20.538507 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:20.538475    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:21.575947 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:21.575903    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:22.781468 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:22.781429    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:22.964265 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:22.964230    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:23.560020 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:23.559990    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:23.924377 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:23.924289    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:24.598415 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:24.598380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:24.692888 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:24.692852    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:27.141271 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:27.141235    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:27.241148 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:27.241108    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:28.292250 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:28.292213    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:28.393314 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:28.393281    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:30.155307 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:30.155278    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:30.255380 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:30.255343    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:32.979320 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:32.979284    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt"]
Sep 16 18:53:32.983027 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:32.982990    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:32.984222 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:32.984205    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt"
Sep 16 18:53:32.987117 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:32.987093    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 18:53:32.987269 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:32.987135    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 18:53:32.987269 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:32.987160    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:53:32.988506 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:32.988472    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:53:32.988741 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:32.988719    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-r856j\""
Sep 16 18:53:32.990640 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:32.990617    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt"]
Sep 16 18:53:33.133350 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:33.133309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6njzf\" (UniqueName: \"kubernetes.io/projected/0e1df62f-f49b-4f8e-bdf4-0502e635115d-kube-api-access-6njzf\") pod \"kuadrant-console-plugin-fbdd4f445-nqdvt\" (UID: \"0e1df62f-f49b-4f8e-bdf4-0502e635115d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt"
Sep 16 18:53:33.133532 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:33.133365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0e1df62f-f49b-4f8e-bdf4-0502e635115d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nqdvt\" (UID: \"0e1df62f-f49b-4f8e-bdf4-0502e635115d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt"
Sep 16 18:53:33.133532 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:33.133407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e1df62f-f49b-4f8e-bdf4-0502e635115d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nqdvt\" (UID: \"0e1df62f-f49b-4f8e-bdf4-0502e635115d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt"
Sep 16 18:53:33.191425 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:33.191384    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:33.234595 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:33.234488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6njzf\" (UniqueName: \"kubernetes.io/projected/0e1df62f-f49b-4f8e-bdf4-0502e635115d-kube-api-access-6njzf\") pod \"kuadrant-console-plugin-fbdd4f445-nqdvt\" (UID: \"0e1df62f-f49b-4f8e-bdf4-0502e635115d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt"
Sep 16 18:53:33.234595 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:33.234544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0e1df62f-f49b-4f8e-bdf4-0502e635115d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nqdvt\" (UID: \"0e1df62f-f49b-4f8e-bdf4-0502e635115d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt"
Sep 16 18:53:33.234595 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:33.234570    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e1df62f-f49b-4f8e-bdf4-0502e635115d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nqdvt\" (UID: \"0e1df62f-f49b-4f8e-bdf4-0502e635115d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt"
Sep 16 18:53:33.235314 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:33.235290    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0e1df62f-f49b-4f8e-bdf4-0502e635115d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nqdvt\" (UID: \"0e1df62f-f49b-4f8e-bdf4-0502e635115d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt"
Sep 16 18:53:33.237196 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:33.237169    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e1df62f-f49b-4f8e-bdf4-0502e635115d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nqdvt\" (UID: \"0e1df62f-f49b-4f8e-bdf4-0502e635115d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt"
Sep 16 18:53:33.243317 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:33.243293    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6njzf\" (UniqueName: \"kubernetes.io/projected/0e1df62f-f49b-4f8e-bdf4-0502e635115d-kube-api-access-6njzf\") pod \"kuadrant-console-plugin-fbdd4f445-nqdvt\" (UID: \"0e1df62f-f49b-4f8e-bdf4-0502e635115d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt"
Sep 16 18:53:33.294282 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:33.294242    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt"
Sep 16 18:53:33.570751 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:33.570686    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt"]
Sep 16 18:53:33.573160 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:53:33.573128    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e1df62f_f49b_4f8e_bdf4_0502e635115d.slice/crio-440c37c9c43d296c156f71f2d6c63130309eec9b32ae9ef7703e2ac8b6d6e3f4 WatchSource:0}: Error finding container 440c37c9c43d296c156f71f2d6c63130309eec9b32ae9ef7703e2ac8b6d6e3f4: Status 404 returned error can't find the container with id 440c37c9c43d296c156f71f2d6c63130309eec9b32ae9ef7703e2ac8b6d6e3f4
Sep 16 18:53:33.932345 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:33.932256    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt" event={"ID":"0e1df62f-f49b-4f8e-bdf4-0502e635115d","Type":"ContainerStarted","Data":"440c37c9c43d296c156f71f2d6c63130309eec9b32ae9ef7703e2ac8b6d6e3f4"}
Sep 16 18:53:35.349979 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:35.349869    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:35.450003 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:35.449968    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:35.556764 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:35.556728    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:35.665566 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:35.665489    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:38.965024 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:38.964981    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt" event={"ID":"0e1df62f-f49b-4f8e-bdf4-0502e635115d","Type":"ContainerStarted","Data":"a78d3435c6c304e42d8d92505635c6f840cbda085aea36a92b858081b92bb0a5"}
Sep 16 18:53:38.985833 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:38.985785    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nqdvt" podStartSLOduration=2.431474887 podStartE2EDuration="6.985769777s" totalImagesPullingTime="4.55429489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:38.984797992 +0000 UTC m=+476.815406538" watchObservedRunningTime="2026-09-16 18:53:38.985769777 +0000 UTC m=+476.816378318"
Sep 16 18:53:39.768913 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:39.768876    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:39.869869 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:39.869828    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:39.936435 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:39.936401    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:46.746087 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:46.746042    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:46.851765 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:46.851738    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:49.357783 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:49.357742    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:49.453858 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:49.453831    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:52.557888 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:52.557851    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:54.744828 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:54.744791    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:54.799775 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:54.799748    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:55.140021 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:55.139996    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:55.240004 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:55.239976    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:59.364924 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:59.364855    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:59.466061 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:59.465998    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:59.565383 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:53:59.565298    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:02.206000 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:02.205947    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:06.401216 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:06.401166    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:06.504788 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:06.504754    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:16.765100 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:16.765055    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:16.864024 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:16.863967    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:16.945335 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:16.945285    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:17.588802 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.588765    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jc8x8"]
Sep 16 18:54:17.610258 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.610219    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jc8x8"]
Sep 16 18:54:17.610465 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.610351    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jc8x8"
Sep 16 18:54:17.613555 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.613529    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-blz4m\""
Sep 16 18:54:17.617221 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.617188    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:17.691328 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.691272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpkgn\" (UniqueName: \"kubernetes.io/projected/dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1-kube-api-access-qpkgn\") pod \"authorino-7bcfd8f9cd-jc8x8\" (UID: \"dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jc8x8"
Sep 16 18:54:17.693351 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.693323    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-mpz2d"]
Sep 16 18:54:17.696367 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.696347    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mpz2d"
Sep 16 18:54:17.703651 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.703589    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mpz2d"]
Sep 16 18:54:17.721234 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.721201    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:17.792290 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.792247    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw559\" (UniqueName: \"kubernetes.io/projected/bb1993ab-35a4-455a-b08f-3a2d9b94dddd-kube-api-access-jw559\") pod \"authorino-59c974bd48-mpz2d\" (UID: \"bb1993ab-35a4-455a-b08f-3a2d9b94dddd\") " pod="kuadrant-system/authorino-59c974bd48-mpz2d"
Sep 16 18:54:17.792290 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.792294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpkgn\" (UniqueName: \"kubernetes.io/projected/dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1-kube-api-access-qpkgn\") pod \"authorino-7bcfd8f9cd-jc8x8\" (UID: \"dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jc8x8"
Sep 16 18:54:17.802997 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.802957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpkgn\" (UniqueName: \"kubernetes.io/projected/dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1-kube-api-access-qpkgn\") pod \"authorino-7bcfd8f9cd-jc8x8\" (UID: \"dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jc8x8"
Sep 16 18:54:17.893595 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.893490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw559\" (UniqueName: \"kubernetes.io/projected/bb1993ab-35a4-455a-b08f-3a2d9b94dddd-kube-api-access-jw559\") pod \"authorino-59c974bd48-mpz2d\" (UID: \"bb1993ab-35a4-455a-b08f-3a2d9b94dddd\") " pod="kuadrant-system/authorino-59c974bd48-mpz2d"
Sep 16 18:54:17.902826 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.902796    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw559\" (UniqueName: \"kubernetes.io/projected/bb1993ab-35a4-455a-b08f-3a2d9b94dddd-kube-api-access-jw559\") pod \"authorino-59c974bd48-mpz2d\" (UID: \"bb1993ab-35a4-455a-b08f-3a2d9b94dddd\") " pod="kuadrant-system/authorino-59c974bd48-mpz2d"
Sep 16 18:54:17.920790 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:17.920754    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jc8x8"
Sep 16 18:54:18.006898 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:18.006860    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mpz2d"
Sep 16 18:54:18.211919 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:18.211837    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jc8x8"]
Sep 16 18:54:18.217811 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:54:18.217777    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcf9eb4d_9c60_4ab0_86fb_3bb8313471b1.slice/crio-3bb0b238e562a400530a78115d57543982f15765f17b28dfa33090e7f0c72502 WatchSource:0}: Error finding container 3bb0b238e562a400530a78115d57543982f15765f17b28dfa33090e7f0c72502: Status 404 returned error can't find the container with id 3bb0b238e562a400530a78115d57543982f15765f17b28dfa33090e7f0c72502
Sep 16 18:54:18.277851 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:18.277817    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mpz2d"]
Sep 16 18:54:18.280922 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:54:18.280892    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb1993ab_35a4_455a_b08f_3a2d9b94dddd.slice/crio-003f6677b6a5010ad619b495114a3439c6ff1e0d81312395d54278ec6b8498c0 WatchSource:0}: Error finding container 003f6677b6a5010ad619b495114a3439c6ff1e0d81312395d54278ec6b8498c0: Status 404 returned error can't find the container with id 003f6677b6a5010ad619b495114a3439c6ff1e0d81312395d54278ec6b8498c0
Sep 16 18:54:19.144025 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:19.143966    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mpz2d" event={"ID":"bb1993ab-35a4-455a-b08f-3a2d9b94dddd","Type":"ContainerStarted","Data":"003f6677b6a5010ad619b495114a3439c6ff1e0d81312395d54278ec6b8498c0"}
Sep 16 18:54:19.145510 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:19.145471    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jc8x8" event={"ID":"dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1","Type":"ContainerStarted","Data":"3bb0b238e562a400530a78115d57543982f15765f17b28dfa33090e7f0c72502"}
Sep 16 18:54:21.158322 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:21.158273    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jc8x8" event={"ID":"dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1","Type":"ContainerStarted","Data":"6469494f6c0acb24bf7b2983d2492a59db8f3bb64b30afb6ed1287ba68f0b095"}
Sep 16 18:54:21.174301 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:21.174237    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-jc8x8" podStartSLOduration=2.053508473 podStartE2EDuration="4.174218486s" totalImagesPullingTime="2.120710013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:21.174177983 +0000 UTC m=+519.004786528" watchObservedRunningTime="2026-09-16 18:54:21.174218486 +0000 UTC m=+519.004827210"
Sep 16 18:54:23.170326 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:23.170283    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mpz2d" event={"ID":"bb1993ab-35a4-455a-b08f-3a2d9b94dddd","Type":"ContainerStarted","Data":"5a5e8d0168dfd27946c25e050db8c59ce4e382fb857b3e36fc4c71d03cedd248"}
Sep 16 18:54:23.209491 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:23.209413    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-mpz2d" podStartSLOduration=1.4119186479999999 podStartE2EDuration="6.209392535s" totalImagesPullingTime="4.797473887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:23.185536308 +0000 UTC m=+521.016144854" watchObservedRunningTime="2026-09-16 18:54:23.209392535 +0000 UTC m=+521.040001098"
Sep 16 18:54:23.210697 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:23.210670    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jc8x8"]
Sep 16 18:54:23.210992 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:23.210968    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-jc8x8" podUID="dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1" containerName="authorino" containerID="cri-o://6469494f6c0acb24bf7b2983d2492a59db8f3bb64b30afb6ed1287ba68f0b095" gracePeriod=30
Sep 16 18:54:23.461736 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:23.461709    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jc8x8"
Sep 16 18:54:23.548894 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:23.548864    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpkgn\" (UniqueName: \"kubernetes.io/projected/dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1-kube-api-access-qpkgn\") pod \"dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1\" (UID: \"dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1\") "
Sep 16 18:54:23.551444 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:23.551413    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1-kube-api-access-qpkgn" (OuterVolumeSpecName: "kube-api-access-qpkgn") pod "dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1" (UID: "dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1"). InnerVolumeSpecName "kube-api-access-qpkgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:23.650290 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:23.650188    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpkgn\" (UniqueName: \"kubernetes.io/projected/dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1-kube-api-access-qpkgn\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:54:24.176216 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:24.176172    2582 generic.go:356] "Generic (PLEG): container finished" podID="dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1" containerID="6469494f6c0acb24bf7b2983d2492a59db8f3bb64b30afb6ed1287ba68f0b095" exitCode=0
Sep 16 18:54:24.176216 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:24.176223    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jc8x8"
Sep 16 18:54:24.176789 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:24.176269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jc8x8" event={"ID":"dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1","Type":"ContainerDied","Data":"6469494f6c0acb24bf7b2983d2492a59db8f3bb64b30afb6ed1287ba68f0b095"}
Sep 16 18:54:24.176789 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:24.176309    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jc8x8" event={"ID":"dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1","Type":"ContainerDied","Data":"3bb0b238e562a400530a78115d57543982f15765f17b28dfa33090e7f0c72502"}
Sep 16 18:54:24.176789 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:24.176325    2582 scope.go:117] "RemoveContainer" containerID="6469494f6c0acb24bf7b2983d2492a59db8f3bb64b30afb6ed1287ba68f0b095"
Sep 16 18:54:24.185929 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:24.185911    2582 scope.go:117] "RemoveContainer" containerID="6469494f6c0acb24bf7b2983d2492a59db8f3bb64b30afb6ed1287ba68f0b095"
Sep 16 18:54:24.186302 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:54:24.186281    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6469494f6c0acb24bf7b2983d2492a59db8f3bb64b30afb6ed1287ba68f0b095\": container with ID starting with 6469494f6c0acb24bf7b2983d2492a59db8f3bb64b30afb6ed1287ba68f0b095 not found: ID does not exist" containerID="6469494f6c0acb24bf7b2983d2492a59db8f3bb64b30afb6ed1287ba68f0b095"
Sep 16 18:54:24.186357 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:24.186318    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6469494f6c0acb24bf7b2983d2492a59db8f3bb64b30afb6ed1287ba68f0b095"} err="failed to get container status \"6469494f6c0acb24bf7b2983d2492a59db8f3bb64b30afb6ed1287ba68f0b095\": rpc error: code = NotFound desc = could not find container \"6469494f6c0acb24bf7b2983d2492a59db8f3bb64b30afb6ed1287ba68f0b095\": container with ID starting with 6469494f6c0acb24bf7b2983d2492a59db8f3bb64b30afb6ed1287ba68f0b095 not found: ID does not exist"
Sep 16 18:54:24.197652 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:24.197616    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jc8x8"]
Sep 16 18:54:24.203725 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:24.203702    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jc8x8"]
Sep 16 18:54:24.704002 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:24.703967    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1" path="/var/lib/kubelet/pods/dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1/volumes"
Sep 16 18:54:36.593947 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:36.593857    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:36.858867 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:36.858786    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:39.193628 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.193594    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-nhjkq"]
Sep 16 18:54:39.194035 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.193946    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1" containerName="authorino"
Sep 16 18:54:39.194035 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.193959    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1" containerName="authorino"
Sep 16 18:54:39.194035 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.194031    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf9eb4d-9c60-4ab0-86fb-3bb8313471b1" containerName="authorino"
Sep 16 18:54:39.196424 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.196407    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-nhjkq"
Sep 16 18:54:39.199724 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.199703    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 18:54:39.205507 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.205483    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-nhjkq"]
Sep 16 18:54:39.295898 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.295860    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7d5f\" (UniqueName: \"kubernetes.io/projected/ed9ecc5c-c5f2-46bd-96ff-c664fae2347c-kube-api-access-g7d5f\") pod \"authorino-6d9898bd99-nhjkq\" (UID: \"ed9ecc5c-c5f2-46bd-96ff-c664fae2347c\") " pod="kuadrant-system/authorino-6d9898bd99-nhjkq"
Sep 16 18:54:39.296086 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.295938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ed9ecc5c-c5f2-46bd-96ff-c664fae2347c-tls-cert\") pod \"authorino-6d9898bd99-nhjkq\" (UID: \"ed9ecc5c-c5f2-46bd-96ff-c664fae2347c\") " pod="kuadrant-system/authorino-6d9898bd99-nhjkq"
Sep 16 18:54:39.397234 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.397202    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ed9ecc5c-c5f2-46bd-96ff-c664fae2347c-tls-cert\") pod \"authorino-6d9898bd99-nhjkq\" (UID: \"ed9ecc5c-c5f2-46bd-96ff-c664fae2347c\") " pod="kuadrant-system/authorino-6d9898bd99-nhjkq"
Sep 16 18:54:39.397404 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.397292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7d5f\" (UniqueName: \"kubernetes.io/projected/ed9ecc5c-c5f2-46bd-96ff-c664fae2347c-kube-api-access-g7d5f\") pod \"authorino-6d9898bd99-nhjkq\" (UID: \"ed9ecc5c-c5f2-46bd-96ff-c664fae2347c\") " pod="kuadrant-system/authorino-6d9898bd99-nhjkq"
Sep 16 18:54:39.399871 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.399837    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ed9ecc5c-c5f2-46bd-96ff-c664fae2347c-tls-cert\") pod \"authorino-6d9898bd99-nhjkq\" (UID: \"ed9ecc5c-c5f2-46bd-96ff-c664fae2347c\") " pod="kuadrant-system/authorino-6d9898bd99-nhjkq"
Sep 16 18:54:39.406296 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.406257    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7d5f\" (UniqueName: \"kubernetes.io/projected/ed9ecc5c-c5f2-46bd-96ff-c664fae2347c-kube-api-access-g7d5f\") pod \"authorino-6d9898bd99-nhjkq\" (UID: \"ed9ecc5c-c5f2-46bd-96ff-c664fae2347c\") " pod="kuadrant-system/authorino-6d9898bd99-nhjkq"
Sep 16 18:54:39.506104 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.506024    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-nhjkq"
Sep 16 18:54:39.777565 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:39.777536    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-nhjkq"]
Sep 16 18:54:39.780079 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:54:39.780052    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded9ecc5c_c5f2_46bd_96ff_c664fae2347c.slice/crio-b3b11a196a16f67ad0f8f359a72776a3af28ea80b6f02f7d2e08fbf906df4bc8 WatchSource:0}: Error finding container b3b11a196a16f67ad0f8f359a72776a3af28ea80b6f02f7d2e08fbf906df4bc8: Status 404 returned error can't find the container with id b3b11a196a16f67ad0f8f359a72776a3af28ea80b6f02f7d2e08fbf906df4bc8
Sep 16 18:54:40.250764 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:40.250663    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-nhjkq" event={"ID":"ed9ecc5c-c5f2-46bd-96ff-c664fae2347c","Type":"ContainerStarted","Data":"b3b11a196a16f67ad0f8f359a72776a3af28ea80b6f02f7d2e08fbf906df4bc8"}
Sep 16 18:54:41.263963 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:41.263914    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-nhjkq" event={"ID":"ed9ecc5c-c5f2-46bd-96ff-c664fae2347c","Type":"ContainerStarted","Data":"4cdc13567a709a02d329365e94d6cc4f7ec4358654f3d974a582ab5e82f1b2b1"}
Sep 16 18:54:41.280617 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:41.280538    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-nhjkq" podStartSLOduration=1.6683709 podStartE2EDuration="2.280519758s" totalImagesPullingTime="612.148858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:41.279376613 +0000 UTC m=+539.109985159" watchObservedRunningTime="2026-09-16 18:54:41.280519758 +0000 UTC m=+539.111128303"
Sep 16 18:54:41.307899 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:41.307859    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mpz2d"]
Sep 16 18:54:41.308117 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:41.308071    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-mpz2d" podUID="bb1993ab-35a4-455a-b08f-3a2d9b94dddd" containerName="authorino" containerID="cri-o://5a5e8d0168dfd27946c25e050db8c59ce4e382fb857b3e36fc4c71d03cedd248" gracePeriod=30
Sep 16 18:54:41.569925 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:41.569894    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mpz2d"
Sep 16 18:54:41.719452 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:41.719395    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw559\" (UniqueName: \"kubernetes.io/projected/bb1993ab-35a4-455a-b08f-3a2d9b94dddd-kube-api-access-jw559\") pod \"bb1993ab-35a4-455a-b08f-3a2d9b94dddd\" (UID: \"bb1993ab-35a4-455a-b08f-3a2d9b94dddd\") "
Sep 16 18:54:41.721901 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:41.721868    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1993ab-35a4-455a-b08f-3a2d9b94dddd-kube-api-access-jw559" (OuterVolumeSpecName: "kube-api-access-jw559") pod "bb1993ab-35a4-455a-b08f-3a2d9b94dddd" (UID: "bb1993ab-35a4-455a-b08f-3a2d9b94dddd"). InnerVolumeSpecName "kube-api-access-jw559". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:41.820884 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:41.820775    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw559\" (UniqueName: \"kubernetes.io/projected/bb1993ab-35a4-455a-b08f-3a2d9b94dddd-kube-api-access-jw559\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:54:42.270294 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:42.270251    2582 generic.go:356] "Generic (PLEG): container finished" podID="bb1993ab-35a4-455a-b08f-3a2d9b94dddd" containerID="5a5e8d0168dfd27946c25e050db8c59ce4e382fb857b3e36fc4c71d03cedd248" exitCode=0
Sep 16 18:54:42.270823 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:42.270301    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mpz2d"
Sep 16 18:54:42.270823 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:42.270341    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mpz2d" event={"ID":"bb1993ab-35a4-455a-b08f-3a2d9b94dddd","Type":"ContainerDied","Data":"5a5e8d0168dfd27946c25e050db8c59ce4e382fb857b3e36fc4c71d03cedd248"}
Sep 16 18:54:42.270823 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:42.270389    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mpz2d" event={"ID":"bb1993ab-35a4-455a-b08f-3a2d9b94dddd","Type":"ContainerDied","Data":"003f6677b6a5010ad619b495114a3439c6ff1e0d81312395d54278ec6b8498c0"}
Sep 16 18:54:42.270823 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:42.270413    2582 scope.go:117] "RemoveContainer" containerID="5a5e8d0168dfd27946c25e050db8c59ce4e382fb857b3e36fc4c71d03cedd248"
Sep 16 18:54:42.280479 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:42.280456    2582 scope.go:117] "RemoveContainer" containerID="5a5e8d0168dfd27946c25e050db8c59ce4e382fb857b3e36fc4c71d03cedd248"
Sep 16 18:54:42.280824 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:54:42.280798    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a5e8d0168dfd27946c25e050db8c59ce4e382fb857b3e36fc4c71d03cedd248\": container with ID starting with 5a5e8d0168dfd27946c25e050db8c59ce4e382fb857b3e36fc4c71d03cedd248 not found: ID does not exist" containerID="5a5e8d0168dfd27946c25e050db8c59ce4e382fb857b3e36fc4c71d03cedd248"
Sep 16 18:54:42.280946 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:42.280840    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a5e8d0168dfd27946c25e050db8c59ce4e382fb857b3e36fc4c71d03cedd248"} err="failed to get container status \"5a5e8d0168dfd27946c25e050db8c59ce4e382fb857b3e36fc4c71d03cedd248\": rpc error: code = NotFound desc = could not find container \"5a5e8d0168dfd27946c25e050db8c59ce4e382fb857b3e36fc4c71d03cedd248\": container with ID starting with 5a5e8d0168dfd27946c25e050db8c59ce4e382fb857b3e36fc4c71d03cedd248 not found: ID does not exist"
Sep 16 18:54:42.292677 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:42.292641    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mpz2d"]
Sep 16 18:54:42.296297 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:42.296269    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mpz2d"]
Sep 16 18:54:42.704352 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:42.704318    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb1993ab-35a4-455a-b08f-3a2d9b94dddd" path="/var/lib/kubelet/pods/bb1993ab-35a4-455a-b08f-3a2d9b94dddd/volumes"
Sep 16 18:54:44.584485 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:44.584447    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:44.684515 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:44.684481    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:56.731292 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:54:56.731253    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:11.124691 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:55:11.124644    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:11.225784 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:55:11.225735    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:11.300801 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:55:11.300762    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:11.399923 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:55:11.399803    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:11.891251 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:55:11.891209    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:11.990383 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:55:11.990331    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:12.091749 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:55:12.091711    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:13.078903 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:55:13.078869    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:13.139545 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:55:13.139513    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:28.257478 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:56:28.257428    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:29.613987 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:56:29.613929    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:23.890032 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:23.889990    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:28.443991 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.443959    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56d487d596-g74ht"]
Sep 16 18:57:28.444326 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.444291    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb1993ab-35a4-455a-b08f-3a2d9b94dddd" containerName="authorino"
Sep 16 18:57:28.444326 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.444304    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb1993ab-35a4-455a-b08f-3a2d9b94dddd" containerName="authorino"
Sep 16 18:57:28.444399 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.444374    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb1993ab-35a4-455a-b08f-3a2d9b94dddd" containerName="authorino"
Sep 16 18:57:28.447289 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.447274    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.461812 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.461782    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d487d596-g74ht"]
Sep 16 18:57:28.509459 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.509420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8125d5c6-aa86-430f-a13d-f3729eec9839-trusted-ca-bundle\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.509664 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.509467    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8125d5c6-aa86-430f-a13d-f3729eec9839-console-oauth-config\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.509664 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.509491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8125d5c6-aa86-430f-a13d-f3729eec9839-service-ca\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.509664 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.509517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sv4p\" (UniqueName: \"kubernetes.io/projected/8125d5c6-aa86-430f-a13d-f3729eec9839-kube-api-access-9sv4p\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.509664 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.509542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8125d5c6-aa86-430f-a13d-f3729eec9839-oauth-serving-cert\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.509664 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.509641    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8125d5c6-aa86-430f-a13d-f3729eec9839-console-config\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.509838 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.509674    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8125d5c6-aa86-430f-a13d-f3729eec9839-console-serving-cert\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.526484 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.526437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:28.610432 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.610393    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8125d5c6-aa86-430f-a13d-f3729eec9839-trusted-ca-bundle\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.610432 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.610436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8125d5c6-aa86-430f-a13d-f3729eec9839-console-oauth-config\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.610786 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.610455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8125d5c6-aa86-430f-a13d-f3729eec9839-service-ca\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.610786 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.610477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sv4p\" (UniqueName: \"kubernetes.io/projected/8125d5c6-aa86-430f-a13d-f3729eec9839-kube-api-access-9sv4p\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.610786 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.610626    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8125d5c6-aa86-430f-a13d-f3729eec9839-oauth-serving-cert\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.610786 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.610682    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8125d5c6-aa86-430f-a13d-f3729eec9839-console-config\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.610786 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.610716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8125d5c6-aa86-430f-a13d-f3729eec9839-console-serving-cert\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.611348 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.611318    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8125d5c6-aa86-430f-a13d-f3729eec9839-service-ca\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.611539 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.611336    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8125d5c6-aa86-430f-a13d-f3729eec9839-trusted-ca-bundle\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.611654 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.611540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8125d5c6-aa86-430f-a13d-f3729eec9839-console-config\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.611654 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.611380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8125d5c6-aa86-430f-a13d-f3729eec9839-oauth-serving-cert\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.613172 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.613150    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8125d5c6-aa86-430f-a13d-f3729eec9839-console-oauth-config\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.613366 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.613348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8125d5c6-aa86-430f-a13d-f3729eec9839-console-serving-cert\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.633752 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.633711    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sv4p\" (UniqueName: \"kubernetes.io/projected/8125d5c6-aa86-430f-a13d-f3729eec9839-kube-api-access-9sv4p\") pod \"console-56d487d596-g74ht\" (UID: \"8125d5c6-aa86-430f-a13d-f3729eec9839\") " pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:28.668979 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.668948    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:28.735182 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.735097    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:28.756132 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:28.756086    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:29.030384 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:29.030339    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d487d596-g74ht"]
Sep 16 18:57:29.034407 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:57:29.034380    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8125d5c6_aa86_430f_a13d_f3729eec9839.slice/crio-81c726e56e83db1184d8dfaf936be3a05e9f1caf302d07d88590217168d1080a WatchSource:0}: Error finding container 81c726e56e83db1184d8dfaf936be3a05e9f1caf302d07d88590217168d1080a: Status 404 returned error can't find the container with id 81c726e56e83db1184d8dfaf936be3a05e9f1caf302d07d88590217168d1080a
Sep 16 18:57:29.059548 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:29.059504    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d487d596-g74ht" event={"ID":"8125d5c6-aa86-430f-a13d-f3729eec9839","Type":"ContainerStarted","Data":"81c726e56e83db1184d8dfaf936be3a05e9f1caf302d07d88590217168d1080a"}
Sep 16 18:57:29.250512 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:29.250479    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:30.066416 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:30.066362    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d487d596-g74ht" event={"ID":"8125d5c6-aa86-430f-a13d-f3729eec9839","Type":"ContainerStarted","Data":"cbe53fd9e4c906e2fe2bc88444a8940376fb0f220c13b06c0dafa2d11da77b8e"}
Sep 16 18:57:30.086221 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:30.086136    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56d487d596-g74ht" podStartSLOduration=2.086110667 podStartE2EDuration="2.086110667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:30.084443228 +0000 UTC m=+707.915051773" watchObservedRunningTime="2026-09-16 18:57:30.086110667 +0000 UTC m=+707.916719211"
Sep 16 18:57:35.789278 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:35.789177    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:38.756833 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:38.756795    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:38.757234 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:38.756848    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:38.761748 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:38.761726    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:39.116171 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:39.116134    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56d487d596-g74ht"
Sep 16 18:57:39.165946 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:39.165909    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8675b48b47-v4n6f"]
Sep 16 18:57:39.290642 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:39.290591    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:39.391520 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:39.391406    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:39.459478 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:39.459445    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:56.000949 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:57:56.000897    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:03.909877 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:03.909834    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:04.009139 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.009092    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:04.192335 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.192211    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8675b48b47-v4n6f" podUID="3412f7e2-2655-4ddb-9f46-bbac0b5edeb4" containerName="console" containerID="cri-o://27a85cc365b33dcf9111cb3027a77e22391ffc609c23486df023fd1ee007e8c0" gracePeriod=15
Sep 16 18:58:04.460306 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.460271    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:58:04.555689 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.555645    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-service-ca\") pod \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") "
Sep 16 18:58:04.555689 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.555691    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-trusted-ca-bundle\") pod \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") "
Sep 16 18:58:04.556008 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.555735    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-serving-cert\") pod \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") "
Sep 16 18:58:04.556008 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.555802    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-oauth-config\") pod \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") "
Sep 16 18:58:04.556008 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.555888    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-config\") pod \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") "
Sep 16 18:58:04.556008 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.555951    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-oauth-serving-cert\") pod \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") "
Sep 16 18:58:04.556201 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.556012    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfmsm\" (UniqueName: \"kubernetes.io/projected/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-kube-api-access-mfmsm\") pod \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\" (UID: \"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4\") "
Sep 16 18:58:04.556328 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.556298    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-service-ca" (OuterVolumeSpecName: "service-ca") pod "3412f7e2-2655-4ddb-9f46-bbac0b5edeb4" (UID: "3412f7e2-2655-4ddb-9f46-bbac0b5edeb4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:04.556395 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.556304    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3412f7e2-2655-4ddb-9f46-bbac0b5edeb4" (UID: "3412f7e2-2655-4ddb-9f46-bbac0b5edeb4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:04.556395 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.556363    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-config" (OuterVolumeSpecName: "console-config") pod "3412f7e2-2655-4ddb-9f46-bbac0b5edeb4" (UID: "3412f7e2-2655-4ddb-9f46-bbac0b5edeb4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:04.556516 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.556493    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3412f7e2-2655-4ddb-9f46-bbac0b5edeb4" (UID: "3412f7e2-2655-4ddb-9f46-bbac0b5edeb4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:04.558711 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.558672    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-kube-api-access-mfmsm" (OuterVolumeSpecName: "kube-api-access-mfmsm") pod "3412f7e2-2655-4ddb-9f46-bbac0b5edeb4" (UID: "3412f7e2-2655-4ddb-9f46-bbac0b5edeb4"). InnerVolumeSpecName "kube-api-access-mfmsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:04.558832 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.558783    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3412f7e2-2655-4ddb-9f46-bbac0b5edeb4" (UID: "3412f7e2-2655-4ddb-9f46-bbac0b5edeb4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:04.558877 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.558823    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3412f7e2-2655-4ddb-9f46-bbac0b5edeb4" (UID: "3412f7e2-2655-4ddb-9f46-bbac0b5edeb4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:04.660611 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.658208    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-config\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:58:04.660611 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.658251    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-oauth-serving-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:58:04.660611 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.658275    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfmsm\" (UniqueName: \"kubernetes.io/projected/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-kube-api-access-mfmsm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:58:04.660611 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.658293    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-service-ca\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:58:04.660611 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.658307    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-trusted-ca-bundle\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:58:04.660611 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.658320    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-serving-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:58:04.660611 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:04.658341    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4-console-oauth-config\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 18:58:05.240133 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:05.240090    2582 generic.go:356] "Generic (PLEG): container finished" podID="3412f7e2-2655-4ddb-9f46-bbac0b5edeb4" containerID="27a85cc365b33dcf9111cb3027a77e22391ffc609c23486df023fd1ee007e8c0" exitCode=0
Sep 16 18:58:05.240656 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:05.240170    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8675b48b47-v4n6f"
Sep 16 18:58:05.240656 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:05.240177    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8675b48b47-v4n6f" event={"ID":"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4","Type":"ContainerDied","Data":"27a85cc365b33dcf9111cb3027a77e22391ffc609c23486df023fd1ee007e8c0"}
Sep 16 18:58:05.240656 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:05.240244    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8675b48b47-v4n6f" event={"ID":"3412f7e2-2655-4ddb-9f46-bbac0b5edeb4","Type":"ContainerDied","Data":"e059e768a692b3fb5cd173ceba0384a2c3d760e692e76a2a8ddc7435b87b9a7c"}
Sep 16 18:58:05.240656 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:05.240267    2582 scope.go:117] "RemoveContainer" containerID="27a85cc365b33dcf9111cb3027a77e22391ffc609c23486df023fd1ee007e8c0"
Sep 16 18:58:05.249756 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:05.249724    2582 scope.go:117] "RemoveContainer" containerID="27a85cc365b33dcf9111cb3027a77e22391ffc609c23486df023fd1ee007e8c0"
Sep 16 18:58:05.250082 ip-10-0-142-173 kubenswrapper[2582]: E0916 18:58:05.250056    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a85cc365b33dcf9111cb3027a77e22391ffc609c23486df023fd1ee007e8c0\": container with ID starting with 27a85cc365b33dcf9111cb3027a77e22391ffc609c23486df023fd1ee007e8c0 not found: ID does not exist" containerID="27a85cc365b33dcf9111cb3027a77e22391ffc609c23486df023fd1ee007e8c0"
Sep 16 18:58:05.250131 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:05.250095    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a85cc365b33dcf9111cb3027a77e22391ffc609c23486df023fd1ee007e8c0"} err="failed to get container status \"27a85cc365b33dcf9111cb3027a77e22391ffc609c23486df023fd1ee007e8c0\": rpc error: code = NotFound desc = could not find container \"27a85cc365b33dcf9111cb3027a77e22391ffc609c23486df023fd1ee007e8c0\": container with ID starting with 27a85cc365b33dcf9111cb3027a77e22391ffc609c23486df023fd1ee007e8c0 not found: ID does not exist"
Sep 16 18:58:05.260435 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:05.260406    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8675b48b47-v4n6f"]
Sep 16 18:58:05.264703 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:05.264678    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8675b48b47-v4n6f"]
Sep 16 18:58:06.704173 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:06.704132    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3412f7e2-2655-4ddb-9f46-bbac0b5edeb4" path="/var/lib/kubelet/pods/3412f7e2-2655-4ddb-9f46-bbac0b5edeb4/volumes"
Sep 16 18:58:16.197023 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:16.196990    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:16.291012 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:16.290965    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:22.039281 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:22.039239    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:38.352112 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.352064    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:38.564060 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.564015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:38.730222 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.730120    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:38.829843 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.829800    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:58:38.830239 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.830224    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3412f7e2-2655-4ddb-9f46-bbac0b5edeb4" containerName="console"
Sep 16 18:58:38.830290 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.830244    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3412f7e2-2655-4ddb-9f46-bbac0b5edeb4" containerName="console"
Sep 16 18:58:38.830325 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.830310    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3412f7e2-2655-4ddb-9f46-bbac0b5edeb4" containerName="console"
Sep 16 18:58:38.833458 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.833433    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:38.841357 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.841323    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 18:58:38.841535 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.841365    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 18:58:38.841535 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.841493    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-5nx7z\""
Sep 16 18:58:38.841706 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.841676    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 18:58:38.842672 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.842652    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-ckqk4\""
Sep 16 18:58:38.842791 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.842701    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:58:38.842861 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.842820    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 18:58:38.857473 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.857447    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:58:38.980813 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.980694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:38.980813 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.980763    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:38.980813 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.980800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/236cb147-818a-452f-92e8-2d8ebfbe9863-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:38.981056 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.980822    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcw8\" (UniqueName: \"kubernetes.io/projected/236cb147-818a-452f-92e8-2d8ebfbe9863-kube-api-access-qfcw8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:38.981056 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.980878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:38.981056 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.980911    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/236cb147-818a-452f-92e8-2d8ebfbe9863-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:38.981056 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.980973    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:38.981056 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.981029    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:38.998205 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:38.998175    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:39.081844 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.081797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.082080 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.081871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.082080 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.081897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.082080 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.081928    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.082080 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.081953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/236cb147-818a-452f-92e8-2d8ebfbe9863-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.082080 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.081979    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcw8\" (UniqueName: \"kubernetes.io/projected/236cb147-818a-452f-92e8-2d8ebfbe9863-kube-api-access-qfcw8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.082080 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.082009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.082080 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.082040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/236cb147-818a-452f-92e8-2d8ebfbe9863-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.082368 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.082250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.082368 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.082317    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.082633 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.082566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.082775 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.082643    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.082775 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.082693    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/236cb147-818a-452f-92e8-2d8ebfbe9863-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.084784 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.084760    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.084971 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.084953    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/236cb147-818a-452f-92e8-2d8ebfbe9863-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.093550 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.093518    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcw8\" (UniqueName: \"kubernetes.io/projected/236cb147-818a-452f-92e8-2d8ebfbe9863-kube-api-access-qfcw8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.150815 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.150770    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:58:39.446308 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.446271    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:58:39.449492 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:58:39.449453    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236cb147_818a_452f_92e8_2d8ebfbe9863.slice/crio-454c264a25b546d3bd18e8e29c3dd1552f2d53cafdb7d5ec6a956edc87d80302 WatchSource:0}: Error finding container 454c264a25b546d3bd18e8e29c3dd1552f2d53cafdb7d5ec6a956edc87d80302: Status 404 returned error can't find the container with id 454c264a25b546d3bd18e8e29c3dd1552f2d53cafdb7d5ec6a956edc87d80302
Sep 16 18:58:39.452122 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:39.452098    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:58:40.413440 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:40.413394    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"236cb147-818a-452f-92e8-2d8ebfbe9863","Type":"ContainerStarted","Data":"454c264a25b546d3bd18e8e29c3dd1552f2d53cafdb7d5ec6a956edc87d80302"}
Sep 16 18:58:43.323931 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:43.323905    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 18:58:43.430525 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:43.430480    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"236cb147-818a-452f-92e8-2d8ebfbe9863","Type":"ContainerStarted","Data":"3cec65e3439106fb282213a8c77d87f2fd41c68d359840efd1fb90b17762d8dd"}
Sep 16 18:58:46.448587 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:46.448532    2582 generic.go:356] "Generic (PLEG): container finished" podID="236cb147-818a-452f-92e8-2d8ebfbe9863" containerID="3cec65e3439106fb282213a8c77d87f2fd41c68d359840efd1fb90b17762d8dd" exitCode=0
Sep 16 18:58:46.449091 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:46.448623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"236cb147-818a-452f-92e8-2d8ebfbe9863","Type":"ContainerDied","Data":"3cec65e3439106fb282213a8c77d87f2fd41c68d359840efd1fb90b17762d8dd"}
Sep 16 18:58:50.318826 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:50.318776    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:50.421634 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:50.421561    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:51.522829 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.522790    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:51.689708 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.689641    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"]
Sep 16 18:58:51.696354 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.696195    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"
Sep 16 18:58:51.701242 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.701207    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 18:58:51.704788 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.703773    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 18:58:51.704788 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.703951    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-tdxrk\""
Sep 16 18:58:51.704788 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.704102    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 18:58:51.704788 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.704185    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-h246m\""
Sep 16 18:58:51.706700 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.706662    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"]
Sep 16 18:58:51.787526 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.787319    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:51.817794 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.817739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwgv2\" (UniqueName: \"kubernetes.io/projected/61fd4383-619d-4eb0-a437-5c046e18106d-kube-api-access-kwgv2\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg\" (UID: \"61fd4383-619d-4eb0-a437-5c046e18106d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"
Sep 16 18:58:51.818045 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.817808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fd4383-619d-4eb0-a437-5c046e18106d-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg\" (UID: \"61fd4383-619d-4eb0-a437-5c046e18106d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"
Sep 16 18:58:51.918590 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.918505    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwgv2\" (UniqueName: \"kubernetes.io/projected/61fd4383-619d-4eb0-a437-5c046e18106d-kube-api-access-kwgv2\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg\" (UID: \"61fd4383-619d-4eb0-a437-5c046e18106d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"
Sep 16 18:58:51.918590 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.918595    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fd4383-619d-4eb0-a437-5c046e18106d-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg\" (UID: \"61fd4383-619d-4eb0-a437-5c046e18106d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"
Sep 16 18:58:51.922619 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.922555    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fd4383-619d-4eb0-a437-5c046e18106d-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg\" (UID: \"61fd4383-619d-4eb0-a437-5c046e18106d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"
Sep 16 18:58:51.928770 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:51.928731    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwgv2\" (UniqueName: \"kubernetes.io/projected/61fd4383-619d-4eb0-a437-5c046e18106d-kube-api-access-kwgv2\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg\" (UID: \"61fd4383-619d-4eb0-a437-5c046e18106d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"
Sep 16 18:58:52.013413 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:52.013359    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"
Sep 16 18:58:52.319450 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:52.318900    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"]
Sep 16 18:58:52.329784 ip-10-0-142-173 kubenswrapper[2582]: W0916 18:58:52.329748    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fd4383_619d_4eb0_a437_5c046e18106d.slice/crio-484272de2dcc7d26a9d6b75ce13601ffc2c83081f7d6427554f6d6efa19a3025 WatchSource:0}: Error finding container 484272de2dcc7d26a9d6b75ce13601ffc2c83081f7d6427554f6d6efa19a3025: Status 404 returned error can't find the container with id 484272de2dcc7d26a9d6b75ce13601ffc2c83081f7d6427554f6d6efa19a3025
Sep 16 18:58:52.495263 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:58:52.495217    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg" event={"ID":"61fd4383-619d-4eb0-a437-5c046e18106d","Type":"ContainerStarted","Data":"484272de2dcc7d26a9d6b75ce13601ffc2c83081f7d6427554f6d6efa19a3025"}
Sep 16 18:59:12.054460 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:59:12.054338    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:15.636837 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:59:15.636747    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg" event={"ID":"61fd4383-619d-4eb0-a437-5c046e18106d","Type":"ContainerStarted","Data":"c7796fbf8e89f773b1eb272b9cd5370e5e8beacf8e9336b2a1fade7595b41aa7"}
Sep 16 18:59:15.636837 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:59:15.636814    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"
Sep 16 18:59:15.656136 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:59:15.656075    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg" podStartSLOduration=1.687665099 podStartE2EDuration="24.656056339s" totalImagesPullingTime="22.96839124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:15.654430388 +0000 UTC m=+813.485038936" watchObservedRunningTime="2026-09-16 18:59:15.656056339 +0000 UTC m=+813.486664884"
Sep 16 18:59:16.644753 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:59:16.644708    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"236cb147-818a-452f-92e8-2d8ebfbe9863","Type":"ContainerStarted","Data":"1df429370c90bdddabecbd2dd0efbd6401468991cd83ac501af9782edb078696"}
Sep 16 18:59:16.667817 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:59:16.667758    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=3.05372448 podStartE2EDuration="38.667738969s" totalImagesPullingTime="33.182139967s" totalInitContainerRuntime="2.431874522s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:16.665381313 +0000 UTC m=+814.495989860" watchObservedRunningTime="2026-09-16 18:59:16.667738969 +0000 UTC m=+814.498347513"
Sep 16 18:59:30.611325 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:59:30.611284    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:46.647140 ip-10-0-142-173 kubenswrapper[2582]: I0916 18:59:46.647104    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"
Sep 16 19:00:02.015493 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:02.015451    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg" podUID="61fd4383-619d-4eb0-a437-5c046e18106d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:02.049059 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:02.049031    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"]
Sep 16 19:00:02.246024 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:02.245946    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:02.567821 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:02.567792    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:06.646656 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:06.646617    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg" podUID="61fd4383-619d-4eb0-a437-5c046e18106d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:16.646429 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:16.646386    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg" podUID="61fd4383-619d-4eb0-a437-5c046e18106d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:17.068076 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:17.067998    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg" podUID="61fd4383-619d-4eb0-a437-5c046e18106d" containerName="main" containerID="cri-o://c7796fbf8e89f773b1eb272b9cd5370e5e8beacf8e9336b2a1fade7595b41aa7" gracePeriod=45
Sep 16 19:00:26.647464 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:26.647411    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg" podUID="61fd4383-619d-4eb0-a437-5c046e18106d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:26.647979 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:26.647547    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"
Sep 16 19:00:36.647281 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:36.647188    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg" podUID="61fd4383-619d-4eb0-a437-5c046e18106d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:46.646801 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:46.646756    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg" podUID="61fd4383-619d-4eb0-a437-5c046e18106d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:47.328917 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:47.328825    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"
Sep 16 19:00:47.433662 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:47.433610    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwgv2\" (UniqueName: \"kubernetes.io/projected/61fd4383-619d-4eb0-a437-5c046e18106d-kube-api-access-kwgv2\") pod \"61fd4383-619d-4eb0-a437-5c046e18106d\" (UID: \"61fd4383-619d-4eb0-a437-5c046e18106d\") "
Sep 16 19:00:47.433949 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:47.433700    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fd4383-619d-4eb0-a437-5c046e18106d-tls-certs\") pod \"61fd4383-619d-4eb0-a437-5c046e18106d\" (UID: \"61fd4383-619d-4eb0-a437-5c046e18106d\") "
Sep 16 19:00:47.436418 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:47.436378    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61fd4383-619d-4eb0-a437-5c046e18106d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61fd4383-619d-4eb0-a437-5c046e18106d" (UID: "61fd4383-619d-4eb0-a437-5c046e18106d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:47.436539 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:47.436433    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fd4383-619d-4eb0-a437-5c046e18106d-kube-api-access-kwgv2" (OuterVolumeSpecName: "kube-api-access-kwgv2") pod "61fd4383-619d-4eb0-a437-5c046e18106d" (UID: "61fd4383-619d-4eb0-a437-5c046e18106d"). InnerVolumeSpecName "kube-api-access-kwgv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:47.535164 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:47.535122    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fd4383-619d-4eb0-a437-5c046e18106d-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:00:47.535164 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:47.535158    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwgv2\" (UniqueName: \"kubernetes.io/projected/61fd4383-619d-4eb0-a437-5c046e18106d-kube-api-access-kwgv2\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:00:48.114794 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:48.114757    2582 generic.go:356] "Generic (PLEG): container finished" podID="61fd4383-619d-4eb0-a437-5c046e18106d" containerID="c7796fbf8e89f773b1eb272b9cd5370e5e8beacf8e9336b2a1fade7595b41aa7" exitCode=0
Sep 16 19:00:48.115264 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:48.114816    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"
Sep 16 19:00:48.115264 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:48.114851    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg" event={"ID":"61fd4383-619d-4eb0-a437-5c046e18106d","Type":"ContainerDied","Data":"c7796fbf8e89f773b1eb272b9cd5370e5e8beacf8e9336b2a1fade7595b41aa7"}
Sep 16 19:00:48.115264 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:48.114910    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg" event={"ID":"61fd4383-619d-4eb0-a437-5c046e18106d","Type":"ContainerDied","Data":"484272de2dcc7d26a9d6b75ce13601ffc2c83081f7d6427554f6d6efa19a3025"}
Sep 16 19:00:48.115264 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:48.114936    2582 scope.go:117] "RemoveContainer" containerID="c7796fbf8e89f773b1eb272b9cd5370e5e8beacf8e9336b2a1fade7595b41aa7"
Sep 16 19:00:48.127739 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:48.127715    2582 scope.go:117] "RemoveContainer" containerID="c7796fbf8e89f773b1eb272b9cd5370e5e8beacf8e9336b2a1fade7595b41aa7"
Sep 16 19:00:48.128330 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:00:48.128309    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7796fbf8e89f773b1eb272b9cd5370e5e8beacf8e9336b2a1fade7595b41aa7\": container with ID starting with c7796fbf8e89f773b1eb272b9cd5370e5e8beacf8e9336b2a1fade7595b41aa7 not found: ID does not exist" containerID="c7796fbf8e89f773b1eb272b9cd5370e5e8beacf8e9336b2a1fade7595b41aa7"
Sep 16 19:00:48.128400 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:48.128341    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7796fbf8e89f773b1eb272b9cd5370e5e8beacf8e9336b2a1fade7595b41aa7"} err="failed to get container status \"c7796fbf8e89f773b1eb272b9cd5370e5e8beacf8e9336b2a1fade7595b41aa7\": rpc error: code = NotFound desc = could not find container \"c7796fbf8e89f773b1eb272b9cd5370e5e8beacf8e9336b2a1fade7595b41aa7\": container with ID starting with c7796fbf8e89f773b1eb272b9cd5370e5e8beacf8e9336b2a1fade7595b41aa7 not found: ID does not exist"
Sep 16 19:00:48.140138 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:48.140106    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"]
Sep 16 19:00:48.143899 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:48.143870    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-r7ksg"]
Sep 16 19:00:48.704039 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:48.704004    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61fd4383-619d-4eb0-a437-5c046e18106d" path="/var/lib/kubelet/pods/61fd4383-619d-4eb0-a437-5c046e18106d/volumes"
Sep 16 19:00:52.334957 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:52.334905    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:52.452535 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:52.452488    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:57.205051 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:57.205010    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:57.310461 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:57.310420    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:58.707235 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:00:58.707193    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:04.649190 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:04.649152    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:04.858721 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:04.858681    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:08.248043 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.247999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:08.569556 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.569523    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"]
Sep 16 19:01:08.569887 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.569875    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61fd4383-619d-4eb0-a437-5c046e18106d" containerName="main"
Sep 16 19:01:08.569931 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.569890    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="61fd4383-619d-4eb0-a437-5c046e18106d" containerName="main"
Sep 16 19:01:08.569965 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.569951    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="61fd4383-619d-4eb0-a437-5c046e18106d" containerName="main"
Sep 16 19:01:08.572017 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.572000    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"
Sep 16 19:01:08.578664 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.578631    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 16 19:01:08.579027 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.578669    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 16 19:01:08.579027 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.578673    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-6zhvl\""
Sep 16 19:01:08.579027 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.578726    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 16 19:01:08.579027 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.578764    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-n2j5t\""
Sep 16 19:01:08.592228 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.592204    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"]
Sep 16 19:01:08.619166 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.619123    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:08.643722 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.643681    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7156e62-8959-4e67-955e-0cb716d0f83e-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8\" (UID: \"f7156e62-8959-4e67-955e-0cb716d0f83e\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"
Sep 16 19:01:08.643884 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.643821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxhcp\" (UniqueName: \"kubernetes.io/projected/f7156e62-8959-4e67-955e-0cb716d0f83e-kube-api-access-hxhcp\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8\" (UID: \"f7156e62-8959-4e67-955e-0cb716d0f83e\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"
Sep 16 19:01:08.744348 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.744291    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxhcp\" (UniqueName: \"kubernetes.io/projected/f7156e62-8959-4e67-955e-0cb716d0f83e-kube-api-access-hxhcp\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8\" (UID: \"f7156e62-8959-4e67-955e-0cb716d0f83e\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"
Sep 16 19:01:08.744348 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.744353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7156e62-8959-4e67-955e-0cb716d0f83e-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8\" (UID: \"f7156e62-8959-4e67-955e-0cb716d0f83e\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"
Sep 16 19:01:08.747588 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.747554    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7156e62-8959-4e67-955e-0cb716d0f83e-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8\" (UID: \"f7156e62-8959-4e67-955e-0cb716d0f83e\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"
Sep 16 19:01:08.760221 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.760171    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxhcp\" (UniqueName: \"kubernetes.io/projected/f7156e62-8959-4e67-955e-0cb716d0f83e-kube-api-access-hxhcp\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8\" (UID: \"f7156e62-8959-4e67-955e-0cb716d0f83e\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"
Sep 16 19:01:08.882596 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:08.882448    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"
Sep 16 19:01:09.179935 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:09.179900    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"]
Sep 16 19:01:09.182052 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:01:09.182023    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7156e62_8959_4e67_955e_0cb716d0f83e.slice/crio-9994152ea7637ddf6886858d35836aa39b407e558f8e9c778033280e8947183d WatchSource:0}: Error finding container 9994152ea7637ddf6886858d35836aa39b407e558f8e9c778033280e8947183d: Status 404 returned error can't find the container with id 9994152ea7637ddf6886858d35836aa39b407e558f8e9c778033280e8947183d
Sep 16 19:01:09.226928 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:09.226885    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8" event={"ID":"f7156e62-8959-4e67-955e-0cb716d0f83e","Type":"ContainerStarted","Data":"9994152ea7637ddf6886858d35836aa39b407e558f8e9c778033280e8947183d"}
Sep 16 19:01:10.234146 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:10.234098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8" event={"ID":"f7156e62-8959-4e67-955e-0cb716d0f83e","Type":"ContainerStarted","Data":"372b3e23084fbe2359959c180f78f4b5539f9bdef89f79079850d2bc2abab98a"}
Sep 16 19:01:10.234669 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:10.234246    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"
Sep 16 19:01:10.267465 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:10.267403    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8" podStartSLOduration=2.267381977 podStartE2EDuration="2.267381977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:10.26435941 +0000 UTC m=+928.094967956" watchObservedRunningTime="2026-09-16 19:01:10.267381977 +0000 UTC m=+928.097990522"
Sep 16 19:01:18.006285 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:18.006245    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:19.696973 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:19.696928    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:19.803316 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:19.803279    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:35.798714 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:35.798667    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:01:41.241683 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:41.241654    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"
Sep 16 19:01:42.318413 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:42.318355    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:47.605627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:47.605554    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:47.613964 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:47.613927    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"]
Sep 16 19:01:47.700328 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:47.700285    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:48.388591 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:48.388549    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:50.823720 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:50.823680    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="236cb147-818a-452f-92e8-2d8ebfbe9863" containerName="main" containerID="cri-o://1df429370c90bdddabecbd2dd0efbd6401468991cd83ac501af9782edb078696" gracePeriod=45
Sep 16 19:01:51.242428 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.242308    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8" podUID="f7156e62-8959-4e67-955e-0cb716d0f83e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:51.681261 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.681232    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:01:51.750745 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.750711    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-home\") pod \"236cb147-818a-452f-92e8-2d8ebfbe9863\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") "
Sep 16 19:01:51.750964 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.750772    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-tmp-dir\") pod \"236cb147-818a-452f-92e8-2d8ebfbe9863\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") "
Sep 16 19:01:51.750964 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.750832    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfcw8\" (UniqueName: \"kubernetes.io/projected/236cb147-818a-452f-92e8-2d8ebfbe9863-kube-api-access-qfcw8\") pod \"236cb147-818a-452f-92e8-2d8ebfbe9863\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") "
Sep 16 19:01:51.750964 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.750863    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-kserve-provision-location\") pod \"236cb147-818a-452f-92e8-2d8ebfbe9863\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") "
Sep 16 19:01:51.750964 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.750930    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/236cb147-818a-452f-92e8-2d8ebfbe9863-tls-certs\") pod \"236cb147-818a-452f-92e8-2d8ebfbe9863\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") "
Sep 16 19:01:51.751199 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.751037    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-dshm\") pod \"236cb147-818a-452f-92e8-2d8ebfbe9863\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") "
Sep 16 19:01:51.751199 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.751119    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/236cb147-818a-452f-92e8-2d8ebfbe9863-cabundle-cert\") pod \"236cb147-818a-452f-92e8-2d8ebfbe9863\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") "
Sep 16 19:01:51.751199 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.751147    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-model-cache\") pod \"236cb147-818a-452f-92e8-2d8ebfbe9863\" (UID: \"236cb147-818a-452f-92e8-2d8ebfbe9863\") "
Sep 16 19:01:51.751633 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.751520    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/236cb147-818a-452f-92e8-2d8ebfbe9863-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "236cb147-818a-452f-92e8-2d8ebfbe9863" (UID: "236cb147-818a-452f-92e8-2d8ebfbe9863"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:51.751633 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.751558    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-home" (OuterVolumeSpecName: "home") pod "236cb147-818a-452f-92e8-2d8ebfbe9863" (UID: "236cb147-818a-452f-92e8-2d8ebfbe9863"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:51.751796 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.751746    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-model-cache" (OuterVolumeSpecName: "model-cache") pod "236cb147-818a-452f-92e8-2d8ebfbe9863" (UID: "236cb147-818a-452f-92e8-2d8ebfbe9863"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:51.753799 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.753760    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/236cb147-818a-452f-92e8-2d8ebfbe9863-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "236cb147-818a-452f-92e8-2d8ebfbe9863" (UID: "236cb147-818a-452f-92e8-2d8ebfbe9863"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:51.753925 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.753842    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-dshm" (OuterVolumeSpecName: "dshm") pod "236cb147-818a-452f-92e8-2d8ebfbe9863" (UID: "236cb147-818a-452f-92e8-2d8ebfbe9863"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:51.754742 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.754716    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236cb147-818a-452f-92e8-2d8ebfbe9863-kube-api-access-qfcw8" (OuterVolumeSpecName: "kube-api-access-qfcw8") pod "236cb147-818a-452f-92e8-2d8ebfbe9863" (UID: "236cb147-818a-452f-92e8-2d8ebfbe9863"). InnerVolumeSpecName "kube-api-access-qfcw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:51.763799 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.763769    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "236cb147-818a-452f-92e8-2d8ebfbe9863" (UID: "236cb147-818a-452f-92e8-2d8ebfbe9863"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:51.772103 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.772070    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "236cb147-818a-452f-92e8-2d8ebfbe9863" (UID: "236cb147-818a-452f-92e8-2d8ebfbe9863"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:51.852569 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.852465    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/236cb147-818a-452f-92e8-2d8ebfbe9863-cabundle-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:01:51.852569 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.852500    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:01:51.852569 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.852511    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:01:51.852569 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.852519    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:01:51.852569 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.852527    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfcw8\" (UniqueName: \"kubernetes.io/projected/236cb147-818a-452f-92e8-2d8ebfbe9863-kube-api-access-qfcw8\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:01:51.852569 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.852537    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:01:51.852569 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.852547    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/236cb147-818a-452f-92e8-2d8ebfbe9863-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:01:51.852569 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:51.852556    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/236cb147-818a-452f-92e8-2d8ebfbe9863-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:01:52.455746 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:52.455700    2582 generic.go:356] "Generic (PLEG): container finished" podID="236cb147-818a-452f-92e8-2d8ebfbe9863" containerID="1df429370c90bdddabecbd2dd0efbd6401468991cd83ac501af9782edb078696" exitCode=0
Sep 16 19:01:52.455981 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:52.455762    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:01:52.455981 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:52.455784    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"236cb147-818a-452f-92e8-2d8ebfbe9863","Type":"ContainerDied","Data":"1df429370c90bdddabecbd2dd0efbd6401468991cd83ac501af9782edb078696"}
Sep 16 19:01:52.455981 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:52.455854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"236cb147-818a-452f-92e8-2d8ebfbe9863","Type":"ContainerDied","Data":"454c264a25b546d3bd18e8e29c3dd1552f2d53cafdb7d5ec6a956edc87d80302"}
Sep 16 19:01:52.455981 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:52.455886    2582 scope.go:117] "RemoveContainer" containerID="1df429370c90bdddabecbd2dd0efbd6401468991cd83ac501af9782edb078696"
Sep 16 19:01:52.466832 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:52.466808    2582 scope.go:117] "RemoveContainer" containerID="3cec65e3439106fb282213a8c77d87f2fd41c68d359840efd1fb90b17762d8dd"
Sep 16 19:01:52.476049 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:52.476028    2582 scope.go:117] "RemoveContainer" containerID="1df429370c90bdddabecbd2dd0efbd6401468991cd83ac501af9782edb078696"
Sep 16 19:01:52.476396 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:01:52.476376    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df429370c90bdddabecbd2dd0efbd6401468991cd83ac501af9782edb078696\": container with ID starting with 1df429370c90bdddabecbd2dd0efbd6401468991cd83ac501af9782edb078696 not found: ID does not exist" containerID="1df429370c90bdddabecbd2dd0efbd6401468991cd83ac501af9782edb078696"
Sep 16 19:01:52.476466 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:52.476405    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df429370c90bdddabecbd2dd0efbd6401468991cd83ac501af9782edb078696"} err="failed to get container status \"1df429370c90bdddabecbd2dd0efbd6401468991cd83ac501af9782edb078696\": rpc error: code = NotFound desc = could not find container \"1df429370c90bdddabecbd2dd0efbd6401468991cd83ac501af9782edb078696\": container with ID starting with 1df429370c90bdddabecbd2dd0efbd6401468991cd83ac501af9782edb078696 not found: ID does not exist"
Sep 16 19:01:52.476466 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:52.476424    2582 scope.go:117] "RemoveContainer" containerID="3cec65e3439106fb282213a8c77d87f2fd41c68d359840efd1fb90b17762d8dd"
Sep 16 19:01:52.476771 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:01:52.476751    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cec65e3439106fb282213a8c77d87f2fd41c68d359840efd1fb90b17762d8dd\": container with ID starting with 3cec65e3439106fb282213a8c77d87f2fd41c68d359840efd1fb90b17762d8dd not found: ID does not exist" containerID="3cec65e3439106fb282213a8c77d87f2fd41c68d359840efd1fb90b17762d8dd"
Sep 16 19:01:52.476839 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:52.476779    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cec65e3439106fb282213a8c77d87f2fd41c68d359840efd1fb90b17762d8dd"} err="failed to get container status \"3cec65e3439106fb282213a8c77d87f2fd41c68d359840efd1fb90b17762d8dd\": rpc error: code = NotFound desc = could not find container \"3cec65e3439106fb282213a8c77d87f2fd41c68d359840efd1fb90b17762d8dd\": container with ID starting with 3cec65e3439106fb282213a8c77d87f2fd41c68d359840efd1fb90b17762d8dd not found: ID does not exist"
Sep 16 19:01:52.485818 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:52.485788    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:01:52.489946 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:52.489908    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:01:52.704060 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:01:52.704021    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="236cb147-818a-452f-92e8-2d8ebfbe9863" path="/var/lib/kubelet/pods/236cb147-818a-452f-92e8-2d8ebfbe9863/volumes"
Sep 16 19:02:01.241286 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:01.241236    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8" podUID="f7156e62-8959-4e67-955e-0cb716d0f83e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:02.635669 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:02.635593    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8" podUID="f7156e62-8959-4e67-955e-0cb716d0f83e" containerName="main" containerID="cri-o://372b3e23084fbe2359959c180f78f4b5539f9bdef89f79079850d2bc2abab98a" gracePeriod=45
Sep 16 19:02:09.922016 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:09.921898    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:10.017889 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:10.017840    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:11.241377 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:11.241330    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8" podUID="f7156e62-8959-4e67-955e-0cb716d0f83e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:11.241938 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:11.241459    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"
Sep 16 19:02:16.211958 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:16.211902    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:21.241798 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:21.241759    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8" podUID="f7156e62-8959-4e67-955e-0cb716d0f83e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:21.796839 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:21.796803    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:21.869584 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:21.869548    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:24.055236 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:24.055198    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:24.333694 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:24.333585    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:28.916023 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:28.915972    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:31.241617 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:31.241556    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8" podUID="f7156e62-8959-4e67-955e-0cb716d0f83e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:32.887062 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:32.887036    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"
Sep 16 19:02:32.938000 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:32.937909    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7156e62-8959-4e67-955e-0cb716d0f83e-tls-certs\") pod \"f7156e62-8959-4e67-955e-0cb716d0f83e\" (UID: \"f7156e62-8959-4e67-955e-0cb716d0f83e\") "
Sep 16 19:02:32.938000 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:32.937974    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxhcp\" (UniqueName: \"kubernetes.io/projected/f7156e62-8959-4e67-955e-0cb716d0f83e-kube-api-access-hxhcp\") pod \"f7156e62-8959-4e67-955e-0cb716d0f83e\" (UID: \"f7156e62-8959-4e67-955e-0cb716d0f83e\") "
Sep 16 19:02:32.940516 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:32.940479    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7156e62-8959-4e67-955e-0cb716d0f83e-kube-api-access-hxhcp" (OuterVolumeSpecName: "kube-api-access-hxhcp") pod "f7156e62-8959-4e67-955e-0cb716d0f83e" (UID: "f7156e62-8959-4e67-955e-0cb716d0f83e"). InnerVolumeSpecName "kube-api-access-hxhcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:32.940682 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:32.940533    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7156e62-8959-4e67-955e-0cb716d0f83e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7156e62-8959-4e67-955e-0cb716d0f83e" (UID: "f7156e62-8959-4e67-955e-0cb716d0f83e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:33.038975 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:33.038932    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxhcp\" (UniqueName: \"kubernetes.io/projected/f7156e62-8959-4e67-955e-0cb716d0f83e-kube-api-access-hxhcp\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:02:33.038975 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:33.038974    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7156e62-8959-4e67-955e-0cb716d0f83e-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:02:33.655545 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:33.655508    2582 generic.go:356] "Generic (PLEG): container finished" podID="f7156e62-8959-4e67-955e-0cb716d0f83e" containerID="372b3e23084fbe2359959c180f78f4b5539f9bdef89f79079850d2bc2abab98a" exitCode=0
Sep 16 19:02:33.655796 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:33.655563    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"
Sep 16 19:02:33.655796 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:33.655598    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8" event={"ID":"f7156e62-8959-4e67-955e-0cb716d0f83e","Type":"ContainerDied","Data":"372b3e23084fbe2359959c180f78f4b5539f9bdef89f79079850d2bc2abab98a"}
Sep 16 19:02:33.655796 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:33.655647    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8" event={"ID":"f7156e62-8959-4e67-955e-0cb716d0f83e","Type":"ContainerDied","Data":"9994152ea7637ddf6886858d35836aa39b407e558f8e9c778033280e8947183d"}
Sep 16 19:02:33.655796 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:33.655670    2582 scope.go:117] "RemoveContainer" containerID="372b3e23084fbe2359959c180f78f4b5539f9bdef89f79079850d2bc2abab98a"
Sep 16 19:02:33.665125 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:33.665104    2582 scope.go:117] "RemoveContainer" containerID="372b3e23084fbe2359959c180f78f4b5539f9bdef89f79079850d2bc2abab98a"
Sep 16 19:02:33.665441 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:02:33.665421    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372b3e23084fbe2359959c180f78f4b5539f9bdef89f79079850d2bc2abab98a\": container with ID starting with 372b3e23084fbe2359959c180f78f4b5539f9bdef89f79079850d2bc2abab98a not found: ID does not exist" containerID="372b3e23084fbe2359959c180f78f4b5539f9bdef89f79079850d2bc2abab98a"
Sep 16 19:02:33.665490 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:33.665452    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372b3e23084fbe2359959c180f78f4b5539f9bdef89f79079850d2bc2abab98a"} err="failed to get container status \"372b3e23084fbe2359959c180f78f4b5539f9bdef89f79079850d2bc2abab98a\": rpc error: code = NotFound desc = could not find container \"372b3e23084fbe2359959c180f78f4b5539f9bdef89f79079850d2bc2abab98a\": container with ID starting with 372b3e23084fbe2359959c180f78f4b5539f9bdef89f79079850d2bc2abab98a not found: ID does not exist"
Sep 16 19:02:33.678136 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:33.678104    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"]
Sep 16 19:02:33.684425 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:33.684395    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jkrc8"]
Sep 16 19:02:34.704019 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:34.703983    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7156e62-8959-4e67-955e-0cb716d0f83e" path="/var/lib/kubelet/pods/f7156e62-8959-4e67-955e-0cb716d0f83e/volumes"
Sep 16 19:02:37.643458 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.643384    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:37.741050 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.741022    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:37.846830 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.846799    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:37.892385 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.892357    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"]
Sep 16 19:02:37.892778 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.892760    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7156e62-8959-4e67-955e-0cb716d0f83e" containerName="main"
Sep 16 19:02:37.892778 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.892779    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7156e62-8959-4e67-955e-0cb716d0f83e" containerName="main"
Sep 16 19:02:37.892928 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.892799    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="236cb147-818a-452f-92e8-2d8ebfbe9863" containerName="main"
Sep 16 19:02:37.892928 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.892806    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="236cb147-818a-452f-92e8-2d8ebfbe9863" containerName="main"
Sep 16 19:02:37.892928 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.892832    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="236cb147-818a-452f-92e8-2d8ebfbe9863" containerName="storage-initializer"
Sep 16 19:02:37.892928 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.892841    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="236cb147-818a-452f-92e8-2d8ebfbe9863" containerName="storage-initializer"
Sep 16 19:02:37.892928 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.892924    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="236cb147-818a-452f-92e8-2d8ebfbe9863" containerName="main"
Sep 16 19:02:37.893169 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.892937    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7156e62-8959-4e67-955e-0cb716d0f83e" containerName="main"
Sep 16 19:02:37.896329 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.896272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:37.899420 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.899399    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 19:02:37.899609 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.899459    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 19:02:37.899742 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.899476    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-bgpnb\""
Sep 16 19:02:37.899742 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.899628    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:02:37.909233 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.909213    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"]
Sep 16 19:02:37.935152 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.935128    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:37.974997 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.974962    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:37.974997 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.974995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:37.975219 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.975015    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:37.975219 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.975095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:37.975219 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.975127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:37.975219 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.975157    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:37.975421 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.975262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:37.975421 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.975310    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wthz\" (UniqueName: \"kubernetes.io/projected/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-kube-api-access-5wthz\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:37.975421 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:37.975350    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.004542 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.004515    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:38.076599 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.076542    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.076797 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.076611    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.076797 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.076642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wthz\" (UniqueName: \"kubernetes.io/projected/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-kube-api-access-5wthz\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.076797 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.076668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.076797 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.076713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.077065 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.076786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.077065 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.076832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.077065 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.076878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.077065 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.076901    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.077278 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.077105    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.077278 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.077146    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.077278 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.077183    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.077429 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.077366    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.077429 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.077416    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.079183 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.079161    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.079422 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.079403    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.088961 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.088932    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.089086 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.089054    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wthz\" (UniqueName: \"kubernetes.io/projected/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-kube-api-access-5wthz\") pod \"router-gateway-1-openshift-default-5b4d9458bd-s8zj6\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.207862 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.207777    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:38.483320 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.483249    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"]
Sep 16 19:02:38.486612 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:02:38.486585    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5709f9e7_28fa_4da7_9393_4dc8f72f01e7.slice/crio-5add1bbd139f43c0ceff33fce3ca32965b7b46e7874207cf64ad2f8b414c2f08 WatchSource:0}: Error finding container 5add1bbd139f43c0ceff33fce3ca32965b7b46e7874207cf64ad2f8b414c2f08: Status 404 returned error can't find the container with id 5add1bbd139f43c0ceff33fce3ca32965b7b46e7874207cf64ad2f8b414c2f08
Sep 16 19:02:38.680541 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:38.680505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6" event={"ID":"5709f9e7-28fa-4da7-9393-4dc8f72f01e7","Type":"ContainerStarted","Data":"5add1bbd139f43c0ceff33fce3ca32965b7b46e7874207cf64ad2f8b414c2f08"}
Sep 16 19:02:41.167282 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:41.167232    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:02:41.167663 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:41.167366    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:02:41.167663 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:41.167399    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:02:41.698943 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:41.698901    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6" event={"ID":"5709f9e7-28fa-4da7-9393-4dc8f72f01e7","Type":"ContainerStarted","Data":"68e82840763c990f1d9f9c22acbc30b8d35d57718cae95c3fe6c30dc06756735"}
Sep 16 19:02:41.721942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:41.721884    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6" podStartSLOduration=2.043632723 podStartE2EDuration="4.72186551s" totalImagesPullingTime="2.678232787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:41.72082566 +0000 UTC m=+1019.551434206" watchObservedRunningTime="2026-09-16 19:02:41.72186551 +0000 UTC m=+1019.552474133"
Sep 16 19:02:42.208744 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:42.208703    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:42.214386 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:42.214355    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:42.472921 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:42.472838    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:42.705834 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:42.705801    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:42.706992 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:42.706967    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:02:43.815822 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:43.815775    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:49.563036 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:49.562999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:49.664256 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:49.664213    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:50.309865 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.309833    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"]
Sep 16 19:02:50.313180 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.313160    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.317743 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.317725    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 16 19:02:50.317849 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.317766    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-n47z9\""
Sep 16 19:02:50.322523 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.322498    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"]
Sep 16 19:02:50.387787 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.387745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.387977 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.387805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.387977 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.387862    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.387977 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.387908    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.387977 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.387941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxz2l\" (UniqueName: \"kubernetes.io/projected/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-kube-api-access-lxz2l\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.388121 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.387984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.489287 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.489257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.489484 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.489292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.489484 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.489328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.489484 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.489365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.489484 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.489399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.489484 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.489429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxz2l\" (UniqueName: \"kubernetes.io/projected/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-kube-api-access-lxz2l\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.489941 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.489910    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.489941 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.489937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.490226 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.489959    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.491842 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.491812    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.492034 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.492017    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.501009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.500984    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxz2l\" (UniqueName: \"kubernetes.io/projected/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-kube-api-access-lxz2l\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.519887 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.519856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:50.624556 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.624457    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:50.901173 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:50.901148    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"]
Sep 16 19:02:50.902915 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:02:50.902889    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e321bd_5ebd_4482_9d45_7d12cb0edbf2.slice/crio-c85fcd2d61d8f31b89e85dfbbe1114f290644db6e15b0b08a8e8822951b23bdd WatchSource:0}: Error finding container c85fcd2d61d8f31b89e85dfbbe1114f290644db6e15b0b08a8e8822951b23bdd: Status 404 returned error can't find the container with id c85fcd2d61d8f31b89e85dfbbe1114f290644db6e15b0b08a8e8822951b23bdd
Sep 16 19:02:51.753356 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:51.753321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm" event={"ID":"50e321bd-5ebd-4482-9d45-7d12cb0edbf2","Type":"ContainerStarted","Data":"c85fcd2d61d8f31b89e85dfbbe1114f290644db6e15b0b08a8e8822951b23bdd"}
Sep 16 19:02:52.541284 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:52.541257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:52.642019 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:52.641983    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:52.743244 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:52.743199    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:52.759915 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:52.759818    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm" event={"ID":"50e321bd-5ebd-4482-9d45-7d12cb0edbf2","Type":"ContainerStarted","Data":"8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22"}
Sep 16 19:02:52.780727 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:52.780673    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm" podStartSLOduration=1.172767925 podStartE2EDuration="2.780657637s" totalImagesPullingTime="1.607889712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:52.779820913 +0000 UTC m=+1030.610429459" watchObservedRunningTime="2026-09-16 19:02:52.780657637 +0000 UTC m=+1030.611266163"
Sep 16 19:02:52.842334 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:52.842283    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:52.941882 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:52.941849    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:53.625482 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:53.625425    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:54.904980 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:54.904926    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"]
Sep 16 19:02:54.926308 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:54.926261    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:55.481955 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:55.481921    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:55.788352 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:02:55.788304    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:02:55.788352 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:02:55.788352 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:02:55.788352 ip-10-0-142-173 kubenswrapper[2582]:  > containerID="8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22" cmd=["/bin/sleep","15"]
Sep 16 19:02:55.788792 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:55.788381    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm" message="" err=<
Sep 16 19:02:55.788792 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:02:55.788792 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:02:55.788792 ip-10-0-142-173 kubenswrapper[2582]:  >
Sep 16 19:02:55.788792 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:02:55.788401    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:02:55.788792 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:02:55.788792 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:02:55.788792 ip-10-0-142-173 kubenswrapper[2582]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm" podUID="50e321bd-5ebd-4482-9d45-7d12cb0edbf2" containerName="main" containerID="cri-o://8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22"
Sep 16 19:02:55.788792 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:55.788457    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm" podUID="50e321bd-5ebd-4482-9d45-7d12cb0edbf2" containerName="main" containerID="cri-o://8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22" gracePeriod=60
Sep 16 19:02:56.231811 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.231773    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:56.368969 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.368858    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-home\") pod \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") "
Sep 16 19:02:56.368969 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.368963    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxz2l\" (UniqueName: \"kubernetes.io/projected/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-kube-api-access-lxz2l\") pod \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") "
Sep 16 19:02:56.369180 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.368993    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-dshm\") pod \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") "
Sep 16 19:02:56.369180 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.369045    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-model-cache\") pod \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") "
Sep 16 19:02:56.369180 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.369077    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-tmp-dir\") pod \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") "
Sep 16 19:02:56.369180 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.369159    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-home" (OuterVolumeSpecName: "home") pod "50e321bd-5ebd-4482-9d45-7d12cb0edbf2" (UID: "50e321bd-5ebd-4482-9d45-7d12cb0edbf2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:56.369180 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.369177    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-tls-certs\") pod \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\" (UID: \"50e321bd-5ebd-4482-9d45-7d12cb0edbf2\") "
Sep 16 19:02:56.369391 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.369314    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50e321bd-5ebd-4482-9d45-7d12cb0edbf2" (UID: "50e321bd-5ebd-4482-9d45-7d12cb0edbf2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:56.369391 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.369310    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-model-cache" (OuterVolumeSpecName: "model-cache") pod "50e321bd-5ebd-4482-9d45-7d12cb0edbf2" (UID: "50e321bd-5ebd-4482-9d45-7d12cb0edbf2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:56.369494 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.369475    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:02:56.369543 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.369526    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:02:56.369613 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.369541    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:02:56.371745 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.371713    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-kube-api-access-lxz2l" (OuterVolumeSpecName: "kube-api-access-lxz2l") pod "50e321bd-5ebd-4482-9d45-7d12cb0edbf2" (UID: "50e321bd-5ebd-4482-9d45-7d12cb0edbf2"). InnerVolumeSpecName "kube-api-access-lxz2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:56.371884 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.371712    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-dshm" (OuterVolumeSpecName: "dshm") pod "50e321bd-5ebd-4482-9d45-7d12cb0edbf2" (UID: "50e321bd-5ebd-4482-9d45-7d12cb0edbf2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:56.371884 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.371715    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50e321bd-5ebd-4482-9d45-7d12cb0edbf2" (UID: "50e321bd-5ebd-4482-9d45-7d12cb0edbf2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:56.470068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.470022    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:02:56.470068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.470059    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxz2l\" (UniqueName: \"kubernetes.io/projected/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-kube-api-access-lxz2l\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:02:56.470068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.470070    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e321bd-5ebd-4482-9d45-7d12cb0edbf2-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:02:56.783155 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.783116    2582 generic.go:356] "Generic (PLEG): container finished" podID="50e321bd-5ebd-4482-9d45-7d12cb0edbf2" containerID="8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22" exitCode=0
Sep 16 19:02:56.783339 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.783167    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"
Sep 16 19:02:56.783339 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.783200    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm" event={"ID":"50e321bd-5ebd-4482-9d45-7d12cb0edbf2","Type":"ContainerDied","Data":"8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22"}
Sep 16 19:02:56.783339 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.783262    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm" event={"ID":"50e321bd-5ebd-4482-9d45-7d12cb0edbf2","Type":"ContainerDied","Data":"c85fcd2d61d8f31b89e85dfbbe1114f290644db6e15b0b08a8e8822951b23bdd"}
Sep 16 19:02:56.783339 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.783286    2582 scope.go:117] "RemoveContainer" containerID="8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22"
Sep 16 19:02:56.792726 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.792497    2582 scope.go:117] "RemoveContainer" containerID="8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22"
Sep 16 19:02:56.792882 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:02:56.792861    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22\": container with ID starting with 8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22 not found: ID does not exist" containerID="8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22"
Sep 16 19:02:56.792937 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.792893    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22"} err="failed to get container status \"8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22\": rpc error: code = NotFound desc = could not find container \"8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22\": container with ID starting with 8cf334db3be6f26701a7c329f68f21c5a4cbe5771b5db97d824f2c55dde0ba22 not found: ID does not exist"
Sep 16 19:02:56.802819 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.802768    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"]
Sep 16 19:02:56.807362 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:56.807299    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-bbsdm"]
Sep 16 19:02:57.517333 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:57.517280    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"]
Sep 16 19:02:57.517905 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:57.517562    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6" podUID="5709f9e7-28fa-4da7-9393-4dc8f72f01e7" containerName="istio-proxy" containerID="cri-o://68e82840763c990f1d9f9c22acbc30b8d35d57718cae95c3fe6c30dc06756735" gracePeriod=30
Sep 16 19:02:57.612294 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:57.612238    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:58.704866 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:58.704825    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50e321bd-5ebd-4482-9d45-7d12cb0edbf2" path="/var/lib/kubelet/pods/50e321bd-5ebd-4482-9d45-7d12cb0edbf2/volumes"
Sep 16 19:02:58.747833 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:02:58.747765    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6" podUID="5709f9e7-28fa-4da7-9393-4dc8f72f01e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:03:00.807284 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:00.807233    2582 generic.go:356] "Generic (PLEG): container finished" podID="5709f9e7-28fa-4da7-9393-4dc8f72f01e7" containerID="68e82840763c990f1d9f9c22acbc30b8d35d57718cae95c3fe6c30dc06756735" exitCode=0
Sep 16 19:03:00.807820 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:00.807294    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6" event={"ID":"5709f9e7-28fa-4da7-9393-4dc8f72f01e7","Type":"ContainerDied","Data":"68e82840763c990f1d9f9c22acbc30b8d35d57718cae95c3fe6c30dc06756735"}
Sep 16 19:03:00.874511 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:00.874449    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:03:01.014235 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.014190    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-envoy\") pod \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") "
Sep 16 19:03:01.014436 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.014259    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-token\") pod \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") "
Sep 16 19:03:01.014436 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.014289    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wthz\" (UniqueName: \"kubernetes.io/projected/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-kube-api-access-5wthz\") pod \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") "
Sep 16 19:03:01.014436 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.014332    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-podinfo\") pod \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") "
Sep 16 19:03:01.014436 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.014372    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-credential-socket\") pod \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") "
Sep 16 19:03:01.014436 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.014423    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-workload-certs\") pod \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") "
Sep 16 19:03:01.014748 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.014445    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-workload-socket\") pod \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") "
Sep 16 19:03:01.014748 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.014469    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-data\") pod \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") "
Sep 16 19:03:01.014748 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.014554    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istiod-ca-cert\") pod \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\" (UID: \"5709f9e7-28fa-4da7-9393-4dc8f72f01e7\") "
Sep 16 19:03:01.014904 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.014738    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5709f9e7-28fa-4da7-9393-4dc8f72f01e7" (UID: "5709f9e7-28fa-4da7-9393-4dc8f72f01e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:01.015128 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.014880    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5709f9e7-28fa-4da7-9393-4dc8f72f01e7" (UID: "5709f9e7-28fa-4da7-9393-4dc8f72f01e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:01.015128 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.014969    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5709f9e7-28fa-4da7-9393-4dc8f72f01e7" (UID: "5709f9e7-28fa-4da7-9393-4dc8f72f01e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:01.015128 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.014985    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "5709f9e7-28fa-4da7-9393-4dc8f72f01e7" (UID: "5709f9e7-28fa-4da7-9393-4dc8f72f01e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:01.015128 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.015047    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-credential-socket\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:01.015368 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.015218    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5709f9e7-28fa-4da7-9393-4dc8f72f01e7" (UID: "5709f9e7-28fa-4da7-9393-4dc8f72f01e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:01.016748 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.016710    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5709f9e7-28fa-4da7-9393-4dc8f72f01e7" (UID: "5709f9e7-28fa-4da7-9393-4dc8f72f01e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:01.016748 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.016714    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-kube-api-access-5wthz" (OuterVolumeSpecName: "kube-api-access-5wthz") pod "5709f9e7-28fa-4da7-9393-4dc8f72f01e7" (UID: "5709f9e7-28fa-4da7-9393-4dc8f72f01e7"). InnerVolumeSpecName "kube-api-access-5wthz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:01.016905 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.016773    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "5709f9e7-28fa-4da7-9393-4dc8f72f01e7" (UID: "5709f9e7-28fa-4da7-9393-4dc8f72f01e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:01.016905 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.016886    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5709f9e7-28fa-4da7-9393-4dc8f72f01e7" (UID: "5709f9e7-28fa-4da7-9393-4dc8f72f01e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:03:01.116543 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.116505    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-workload-socket\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:01.116543 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.116533    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-data\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:01.116543 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.116543    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istiod-ca-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:01.116543 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.116552    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-envoy\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:01.116835 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.116560    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-token\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:01.116835 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.116569    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wthz\" (UniqueName: \"kubernetes.io/projected/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-kube-api-access-5wthz\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:01.116835 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.116604    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-istio-podinfo\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:01.116835 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.116612    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5709f9e7-28fa-4da7-9393-4dc8f72f01e7-workload-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:01.813788 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.813762    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"
Sep 16 19:03:01.813788 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.813778    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6" event={"ID":"5709f9e7-28fa-4da7-9393-4dc8f72f01e7","Type":"ContainerDied","Data":"5add1bbd139f43c0ceff33fce3ca32965b7b46e7874207cf64ad2f8b414c2f08"}
Sep 16 19:03:01.814379 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.813845    2582 scope.go:117] "RemoveContainer" containerID="68e82840763c990f1d9f9c22acbc30b8d35d57718cae95c3fe6c30dc06756735"
Sep 16 19:03:01.841118 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.841076    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"]
Sep 16 19:03:01.847328 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:01.847296    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-s8zj6"]
Sep 16 19:03:02.709016 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:02.708980    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5709f9e7-28fa-4da7-9393-4dc8f72f01e7" path="/var/lib/kubelet/pods/5709f9e7-28fa-4da7-9393-4dc8f72f01e7/volumes"
Sep 16 19:03:02.941400 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:02.941363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:03.544492 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:03.544446    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:08.397795 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.397764    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"]
Sep 16 19:03:08.398163 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.398060    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5709f9e7-28fa-4da7-9393-4dc8f72f01e7" containerName="istio-proxy"
Sep 16 19:03:08.398163 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.398070    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5709f9e7-28fa-4da7-9393-4dc8f72f01e7" containerName="istio-proxy"
Sep 16 19:03:08.398163 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.398089    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50e321bd-5ebd-4482-9d45-7d12cb0edbf2" containerName="main"
Sep 16 19:03:08.398163 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.398095    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="50e321bd-5ebd-4482-9d45-7d12cb0edbf2" containerName="main"
Sep 16 19:03:08.398296 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.398167    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5709f9e7-28fa-4da7-9393-4dc8f72f01e7" containerName="istio-proxy"
Sep 16 19:03:08.398296 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.398176    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="50e321bd-5ebd-4482-9d45-7d12cb0edbf2" containerName="main"
Sep 16 19:03:08.400331 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.400316    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.404652 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.404558    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 16 19:03:08.404652 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.404611    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 19:03:08.404652 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.404631    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 19:03:08.404948 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.404632    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-xl5ks\""
Sep 16 19:03:08.410859 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.410840    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"]
Sep 16 19:03:08.428232 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.428210    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:08.581478 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.581451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c271122a-a00c-4f80-8429-132e50aa3a98-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.581614 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.581495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.581614 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.581570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.581742 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.581635    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44jd9\" (UniqueName: \"kubernetes.io/projected/c271122a-a00c-4f80-8429-132e50aa3a98-kube-api-access-44jd9\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.581742 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.581680    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.581742 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.581719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.682705 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.682652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.682705 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.682687    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.682849 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.682707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c271122a-a00c-4f80-8429-132e50aa3a98-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.682849 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.682731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.682849 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.682764    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.682849 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.682784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44jd9\" (UniqueName: \"kubernetes.io/projected/c271122a-a00c-4f80-8429-132e50aa3a98-kube-api-access-44jd9\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.683103 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.683084    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.683158 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.683123    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.683194 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.683157    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.685104 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.685078    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.685332 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.685316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c271122a-a00c-4f80-8429-132e50aa3a98-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.691949 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.691926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44jd9\" (UniqueName: \"kubernetes.io/projected/c271122a-a00c-4f80-8429-132e50aa3a98-kube-api-access-44jd9\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.710733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.710698    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:08.984613 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:08.984587    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"]
Sep 16 19:03:08.987288 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:03:08.987248    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc271122a_a00c_4f80_8429_132e50aa3a98.slice/crio-f1f619da2d5938bc21ac5d8650b03fd3aabfd754056b1c8db2718622641c3ea0 WatchSource:0}: Error finding container f1f619da2d5938bc21ac5d8650b03fd3aabfd754056b1c8db2718622641c3ea0: Status 404 returned error can't find the container with id f1f619da2d5938bc21ac5d8650b03fd3aabfd754056b1c8db2718622641c3ea0
Sep 16 19:03:09.243640 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:09.243548    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:09.851724 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:09.851690    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4" event={"ID":"c271122a-a00c-4f80-8429-132e50aa3a98","Type":"ContainerStarted","Data":"d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9"}
Sep 16 19:03:09.852148 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:09.851742    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4" event={"ID":"c271122a-a00c-4f80-8429-132e50aa3a98","Type":"ContainerStarted","Data":"f1f619da2d5938bc21ac5d8650b03fd3aabfd754056b1c8db2718622641c3ea0"}
Sep 16 19:03:09.873316 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:09.873275    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4" podStartSLOduration=1.8732619910000001 podStartE2EDuration="1.873261991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:09.870541223 +0000 UTC m=+1047.701149767" watchObservedRunningTime="2026-09-16 19:03:09.873261991 +0000 UTC m=+1047.703870536"
Sep 16 19:03:10.064395 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:10.064362    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:10.165279 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:10.165212    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:10.711502 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:10.711474    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:11.427346 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:11.427310    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:11.526734 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:11.526699    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:16.350618 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.350569    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"]
Sep 16 19:03:16.360009 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:03:16.359971    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:03:16.360009 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:03:16.360009 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:03:16.360009 ip-10-0-142-173 kubenswrapper[2582]:  > containerID="d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9" cmd=["/bin/sleep","15"]
Sep 16 19:03:16.360237 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.360020    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4" message="" err=<
Sep 16 19:03:16.360237 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:03:16.360237 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:03:16.360237 ip-10-0-142-173 kubenswrapper[2582]:  >
Sep 16 19:03:16.360237 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:03:16.360031    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:03:16.360237 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:03:16.360237 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:03:16.360237 ip-10-0-142-173 kubenswrapper[2582]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4" podUID="c271122a-a00c-4f80-8429-132e50aa3a98" containerName="main" containerID="cri-o://d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9"
Sep 16 19:03:16.360237 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.360076    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4" podUID="c271122a-a00c-4f80-8429-132e50aa3a98" containerName="main" containerID="cri-o://d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9" gracePeriod=60
Sep 16 19:03:16.443345 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.443311    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:16.596751 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.596725    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:16.742211 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.742106    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-home\") pod \"c271122a-a00c-4f80-8429-132e50aa3a98\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") "
Sep 16 19:03:16.742211 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.742199    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44jd9\" (UniqueName: \"kubernetes.io/projected/c271122a-a00c-4f80-8429-132e50aa3a98-kube-api-access-44jd9\") pod \"c271122a-a00c-4f80-8429-132e50aa3a98\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") "
Sep 16 19:03:16.742463 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.742223    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-dshm\") pod \"c271122a-a00c-4f80-8429-132e50aa3a98\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") "
Sep 16 19:03:16.742463 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.742271    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c271122a-a00c-4f80-8429-132e50aa3a98-tls-certs\") pod \"c271122a-a00c-4f80-8429-132e50aa3a98\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") "
Sep 16 19:03:16.742463 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.742323    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-tmp-dir\") pod \"c271122a-a00c-4f80-8429-132e50aa3a98\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") "
Sep 16 19:03:16.742463 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.742343    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-model-cache\") pod \"c271122a-a00c-4f80-8429-132e50aa3a98\" (UID: \"c271122a-a00c-4f80-8429-132e50aa3a98\") "
Sep 16 19:03:16.742463 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.742435    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-home" (OuterVolumeSpecName: "home") pod "c271122a-a00c-4f80-8429-132e50aa3a98" (UID: "c271122a-a00c-4f80-8429-132e50aa3a98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:16.742789 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.742667    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:16.742789 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.742704    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-model-cache" (OuterVolumeSpecName: "model-cache") pod "c271122a-a00c-4f80-8429-132e50aa3a98" (UID: "c271122a-a00c-4f80-8429-132e50aa3a98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:16.742789 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.742697    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c271122a-a00c-4f80-8429-132e50aa3a98" (UID: "c271122a-a00c-4f80-8429-132e50aa3a98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:16.744755 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.744729    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-dshm" (OuterVolumeSpecName: "dshm") pod "c271122a-a00c-4f80-8429-132e50aa3a98" (UID: "c271122a-a00c-4f80-8429-132e50aa3a98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:16.744868 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.744782    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c271122a-a00c-4f80-8429-132e50aa3a98-kube-api-access-44jd9" (OuterVolumeSpecName: "kube-api-access-44jd9") pod "c271122a-a00c-4f80-8429-132e50aa3a98" (UID: "c271122a-a00c-4f80-8429-132e50aa3a98"). InnerVolumeSpecName "kube-api-access-44jd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:16.744868 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.744789    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c271122a-a00c-4f80-8429-132e50aa3a98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c271122a-a00c-4f80-8429-132e50aa3a98" (UID: "c271122a-a00c-4f80-8429-132e50aa3a98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:16.843772 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.843734    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44jd9\" (UniqueName: \"kubernetes.io/projected/c271122a-a00c-4f80-8429-132e50aa3a98-kube-api-access-44jd9\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:16.843979 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.843916    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:16.843979 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.843949    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c271122a-a00c-4f80-8429-132e50aa3a98-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:16.843979 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.843962    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:16.843979 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.843973    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c271122a-a00c-4f80-8429-132e50aa3a98-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:03:16.881859 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.881829    2582 generic.go:356] "Generic (PLEG): container finished" podID="c271122a-a00c-4f80-8429-132e50aa3a98" containerID="d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9" exitCode=0
Sep 16 19:03:16.881998 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.881877    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"
Sep 16 19:03:16.881998 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.881914    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4" event={"ID":"c271122a-a00c-4f80-8429-132e50aa3a98","Type":"ContainerDied","Data":"d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9"}
Sep 16 19:03:16.881998 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.881973    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4" event={"ID":"c271122a-a00c-4f80-8429-132e50aa3a98","Type":"ContainerDied","Data":"f1f619da2d5938bc21ac5d8650b03fd3aabfd754056b1c8db2718622641c3ea0"}
Sep 16 19:03:16.881998 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.881997    2582 scope.go:117] "RemoveContainer" containerID="d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9"
Sep 16 19:03:16.891749 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.891727    2582 scope.go:117] "RemoveContainer" containerID="d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9"
Sep 16 19:03:16.892121 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:03:16.892094    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9\": container with ID starting with d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9 not found: ID does not exist" containerID="d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9"
Sep 16 19:03:16.892174 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.892131    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9"} err="failed to get container status \"d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9\": rpc error: code = NotFound desc = could not find container \"d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9\": container with ID starting with d874eea997ed81c26861f7386186c32e99475145829998ee0f231bb9a1cdbfc9 not found: ID does not exist"
Sep 16 19:03:16.904956 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.904915    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"]
Sep 16 19:03:16.908827 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:16.908800    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfbtjm4"]
Sep 16 19:03:18.704128 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:18.704092    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c271122a-a00c-4f80-8429-132e50aa3a98" path="/var/lib/kubelet/pods/c271122a-a00c-4f80-8429-132e50aa3a98/volumes"
Sep 16 19:03:20.669139 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:20.669105    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:21.614094 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:21.614053    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:22.838177 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:22.838138    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:27.829834 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:27.829792    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:28.552121 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:28.552070    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:28.619799 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:28.619753    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:31.434965 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.434931    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"]
Sep 16 19:03:31.435385 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.435268    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c271122a-a00c-4f80-8429-132e50aa3a98" containerName="main"
Sep 16 19:03:31.435385 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.435279    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c271122a-a00c-4f80-8429-132e50aa3a98" containerName="main"
Sep 16 19:03:31.435385 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.435341    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c271122a-a00c-4f80-8429-132e50aa3a98" containerName="main"
Sep 16 19:03:31.438473 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.438454    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.441703 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.441676    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:03:31.441703 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.441700    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 19:03:31.441913 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.441752    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 19:03:31.441913 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.441677    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-nqx5d\""
Sep 16 19:03:31.441913 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.441681    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 16 19:03:31.442128 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.442113    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 19:03:31.450960 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.450938    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"]
Sep 16 19:03:31.456092 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.456063    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:31.471174 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.471151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-tls-certs\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.471294 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.471189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-kserve-provision-location\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.471294 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.471209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-home\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.471382 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.471294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-tmp-dir\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.471382 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.471326    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-dshm\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.471382 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.471352    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-model-cache\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.471476 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.471414    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-cabundle-cert\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.471476 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.471445    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvpzr\" (UniqueName: \"kubernetes.io/projected/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-kube-api-access-nvpzr\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.572336 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.572285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-dshm\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.572336 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.572337    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-model-cache\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.572658 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.572377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-cabundle-cert\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.572658 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.572407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpzr\" (UniqueName: \"kubernetes.io/projected/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-kube-api-access-nvpzr\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.572658 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.572464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-tls-certs\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.572658 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.572504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-kserve-provision-location\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.572658 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.572528    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-home\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.572658 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.572624    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-tmp-dir\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.573019 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.572993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-model-cache\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.573088 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.573022    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-tmp-dir\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.573088 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.573026    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-kserve-provision-location\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.573160 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.573148    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-home\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.573520 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.573499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-cabundle-cert\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.575019 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.574996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-dshm\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.575165 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.575149    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-tls-certs\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.582055 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.582030    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvpzr\" (UniqueName: \"kubernetes.io/projected/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-kube-api-access-nvpzr\") pod \"auth-enabled-test-kserve-86c6b664b7-7cgfk\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.710612 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.710513    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"]
Sep 16 19:03:31.715231 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.715210    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"
Sep 16 19:03:31.718119 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.718091    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-tcm5z\""
Sep 16 19:03:31.721847 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.721821    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"]
Sep 16 19:03:31.729015 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.728993    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:31.751127 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.751093    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:31.774476 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.774447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8adc7e9f-6094-4860-9715-deeed44f78dd-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h\" (UID: \"8adc7e9f-6094-4860-9715-deeed44f78dd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"
Sep 16 19:03:31.774649 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.774492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dfxh\" (UniqueName: \"kubernetes.io/projected/8adc7e9f-6094-4860-9715-deeed44f78dd-kube-api-access-5dfxh\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h\" (UID: \"8adc7e9f-6094-4860-9715-deeed44f78dd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"
Sep 16 19:03:31.875443 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.875404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8adc7e9f-6094-4860-9715-deeed44f78dd-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h\" (UID: \"8adc7e9f-6094-4860-9715-deeed44f78dd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"
Sep 16 19:03:31.875443 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.875448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dfxh\" (UniqueName: \"kubernetes.io/projected/8adc7e9f-6094-4860-9715-deeed44f78dd-kube-api-access-5dfxh\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h\" (UID: \"8adc7e9f-6094-4860-9715-deeed44f78dd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"
Sep 16 19:03:31.878216 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.878184    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8adc7e9f-6094-4860-9715-deeed44f78dd-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h\" (UID: \"8adc7e9f-6094-4860-9715-deeed44f78dd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"
Sep 16 19:03:31.884796 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:31.884772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dfxh\" (UniqueName: \"kubernetes.io/projected/8adc7e9f-6094-4860-9715-deeed44f78dd-kube-api-access-5dfxh\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h\" (UID: \"8adc7e9f-6094-4860-9715-deeed44f78dd\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"
Sep 16 19:03:32.028213 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:32.028186    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"
Sep 16 19:03:32.032430 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:32.032410    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"]
Sep 16 19:03:32.034412 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:03:32.034380    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12c8784_dbe5_4995_8b0f_418b6b8ffb7c.slice/crio-2c86fb0d8d272fbf53ff46a0041f49b0981d90e337079bfcdf62ee1be7a702bb WatchSource:0}: Error finding container 2c86fb0d8d272fbf53ff46a0041f49b0981d90e337079bfcdf62ee1be7a702bb: Status 404 returned error can't find the container with id 2c86fb0d8d272fbf53ff46a0041f49b0981d90e337079bfcdf62ee1be7a702bb
Sep 16 19:03:32.306527 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:32.306494    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"]
Sep 16 19:03:32.308409 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:03:32.308379    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8adc7e9f_6094_4860_9715_deeed44f78dd.slice/crio-f78ad11df7b1b4eb5e9056a39c5f4e5f9d4850f148bda8240959cb9a22101b99 WatchSource:0}: Error finding container f78ad11df7b1b4eb5e9056a39c5f4e5f9d4850f148bda8240959cb9a22101b99: Status 404 returned error can't find the container with id f78ad11df7b1b4eb5e9056a39c5f4e5f9d4850f148bda8240959cb9a22101b99
Sep 16 19:03:32.962846 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:32.962798    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h" event={"ID":"8adc7e9f-6094-4860-9715-deeed44f78dd","Type":"ContainerStarted","Data":"a03a6e8ab877d2c6bdd323b45fb56cd8f9a33ea5bb9b4522967ef498c3f1a5f3"}
Sep 16 19:03:32.963367 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:32.962916    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h" event={"ID":"8adc7e9f-6094-4860-9715-deeed44f78dd","Type":"ContainerStarted","Data":"f78ad11df7b1b4eb5e9056a39c5f4e5f9d4850f148bda8240959cb9a22101b99"}
Sep 16 19:03:32.963367 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:32.962953    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"
Sep 16 19:03:32.964610 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:32.964557    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" event={"ID":"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c","Type":"ContainerStarted","Data":"57ff5d60fa773c82bde5d7e26f5458520a53413c6c46e11c2afa4cc851068bc7"}
Sep 16 19:03:32.964723 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:32.964639    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" event={"ID":"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c","Type":"ContainerStarted","Data":"2c86fb0d8d272fbf53ff46a0041f49b0981d90e337079bfcdf62ee1be7a702bb"}
Sep 16 19:03:32.981980 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:32.981926    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h" podStartSLOduration=1.981908014 podStartE2EDuration="1.981908014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:32.980109596 +0000 UTC m=+1070.810718143" watchObservedRunningTime="2026-09-16 19:03:32.981908014 +0000 UTC m=+1070.812516555"
Sep 16 19:03:33.971617 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:33.971558    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:34.067406 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:34.067375    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:34.976859 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:34.976775    2582 generic.go:356] "Generic (PLEG): container finished" podID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerID="57ff5d60fa773c82bde5d7e26f5458520a53413c6c46e11c2afa4cc851068bc7" exitCode=0
Sep 16 19:03:34.977231 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:34.976852    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" event={"ID":"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c","Type":"ContainerDied","Data":"57ff5d60fa773c82bde5d7e26f5458520a53413c6c46e11c2afa4cc851068bc7"}
Sep 16 19:03:35.984501 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:35.984469    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" event={"ID":"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c","Type":"ContainerStarted","Data":"4ec314ed4c9c8de17440a9ff15a8076f700c8680e81d14b7680bf7fd4ee2108a"}
Sep 16 19:03:36.012182 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:36.012124    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podStartSLOduration=2.529927958 podStartE2EDuration="5.012106806s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.482178848s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:36.009110238 +0000 UTC m=+1073.839718783" watchObservedRunningTime="2026-09-16 19:03:36.012106806 +0000 UTC m=+1073.842715350"
Sep 16 19:03:36.751178 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:36.751149    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:41.752213 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:41.752162    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:03:41.753911 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:41.753881    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:03:44.720238 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:44.720201    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:51.752588 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:03:51.752515    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:04:01.751752 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:01.751703    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:04:03.973655 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:03.973616    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"
Sep 16 19:04:11.751824 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:11.751771    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:04:21.752230 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:21.752167    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:04:22.411191 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:22.411146    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:22.507072 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:22.507031    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:25.204540 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:25.204475    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:25.421238 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:25.421193    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:28.586938 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:28.586888    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:28.645536 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:28.645496    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:31.752049 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:31.751980    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:04:34.262424 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:34.262371    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:34.325627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:34.325548    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:41.752611 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:41.752527    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:04:51.752017 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:51.751953    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:04:59.316193 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:59.316109    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:59.414320 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:04:59.414275    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:01.751946 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:01.751900    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:05:11.752360 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:11.752237    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:05:21.752037 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:21.751977    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:05:31.752026 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:31.751962    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:05:41.762454 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:41.762418    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:05:41.770930 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:41.770889    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:05:47.559403 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:47.559362    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"]
Sep 16 19:05:47.567566 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:47.567529    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:47.568024 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:47.567999    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"]
Sep 16 19:05:47.638901 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:47.638864    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:48.911551 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:48.911494    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:49.014569 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:49.014528    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:53.973010 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:53.972969    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h" podUID="8adc7e9f-6094-4860-9715-deeed44f78dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:55.070323 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:55.070279    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:57.643764 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:57.643731    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:57.916130 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:05:57.916046    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:01.001767 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:01.001731    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:01.078451 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:01.078420    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:02.584231 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:02.584178    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main" containerID="cri-o://4ec314ed4c9c8de17440a9ff15a8076f700c8680e81d14b7680bf7fd4ee2108a" gracePeriod=45
Sep 16 19:06:02.588135 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:02.588100    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h" podUID="8adc7e9f-6094-4860-9715-deeed44f78dd" containerName="main" containerID="cri-o://a03a6e8ab877d2c6bdd323b45fb56cd8f9a33ea5bb9b4522967ef498c3f1a5f3" gracePeriod=45
Sep 16 19:06:02.763802 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:02.763756    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 16 19:06:03.352087 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.352057    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:06:03.441203 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.441091    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-kserve-provision-location\") pod \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") "
Sep 16 19:06:03.441203 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.441134    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-cabundle-cert\") pod \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") "
Sep 16 19:06:03.441203 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.441155    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-model-cache\") pod \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") "
Sep 16 19:06:03.441203 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.441173    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvpzr\" (UniqueName: \"kubernetes.io/projected/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-kube-api-access-nvpzr\") pod \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") "
Sep 16 19:06:03.441203 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.441198    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-dshm\") pod \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") "
Sep 16 19:06:03.441709 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.441234    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-tls-certs\") pod \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") "
Sep 16 19:06:03.441709 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.441267    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-tmp-dir\") pod \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") "
Sep 16 19:06:03.441709 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.441362    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-home\") pod \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\" (UID: \"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c\") "
Sep 16 19:06:03.441709 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.441617    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" (UID: "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:03.441709 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.441674    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" (UID: "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:06:03.441968 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.441795    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-cabundle-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:06:03.441968 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.441815    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:06:03.442482 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.442454    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-home" (OuterVolumeSpecName: "home") pod "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" (UID: "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:03.444031 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.444004    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-dshm" (OuterVolumeSpecName: "dshm") pod "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" (UID: "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:03.444163 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.444054    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" (UID: "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:03.444163 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.444144    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-kube-api-access-nvpzr" (OuterVolumeSpecName: "kube-api-access-nvpzr") pod "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" (UID: "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c"). InnerVolumeSpecName "kube-api-access-nvpzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:03.454527 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.454493    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" (UID: "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:03.471080 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.471037    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" (UID: "d12c8784-dbe5-4995-8b0f-418b6b8ffb7c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:03.542248 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.542208    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:06:03.542248 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.542241    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:06:03.542248 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.542252    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:06:03.542485 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.542262    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvpzr\" (UniqueName: \"kubernetes.io/projected/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-kube-api-access-nvpzr\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:06:03.542485 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.542271    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:06:03.542485 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.542280    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:06:03.777733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.777699    2582 generic.go:356] "Generic (PLEG): container finished" podID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerID="4ec314ed4c9c8de17440a9ff15a8076f700c8680e81d14b7680bf7fd4ee2108a" exitCode=0
Sep 16 19:06:03.778200 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.777764    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"
Sep 16 19:06:03.778200 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.777779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" event={"ID":"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c","Type":"ContainerDied","Data":"4ec314ed4c9c8de17440a9ff15a8076f700c8680e81d14b7680bf7fd4ee2108a"}
Sep 16 19:06:03.778200 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.777834    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk" event={"ID":"d12c8784-dbe5-4995-8b0f-418b6b8ffb7c","Type":"ContainerDied","Data":"2c86fb0d8d272fbf53ff46a0041f49b0981d90e337079bfcdf62ee1be7a702bb"}
Sep 16 19:06:03.778200 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.777860    2582 scope.go:117] "RemoveContainer" containerID="4ec314ed4c9c8de17440a9ff15a8076f700c8680e81d14b7680bf7fd4ee2108a"
Sep 16 19:06:03.794086 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.794065    2582 scope.go:117] "RemoveContainer" containerID="57ff5d60fa773c82bde5d7e26f5458520a53413c6c46e11c2afa4cc851068bc7"
Sep 16 19:06:03.802515 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.802486    2582 scope.go:117] "RemoveContainer" containerID="4ec314ed4c9c8de17440a9ff15a8076f700c8680e81d14b7680bf7fd4ee2108a"
Sep 16 19:06:03.802842 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:06:03.802822    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec314ed4c9c8de17440a9ff15a8076f700c8680e81d14b7680bf7fd4ee2108a\": container with ID starting with 4ec314ed4c9c8de17440a9ff15a8076f700c8680e81d14b7680bf7fd4ee2108a not found: ID does not exist" containerID="4ec314ed4c9c8de17440a9ff15a8076f700c8680e81d14b7680bf7fd4ee2108a"
Sep 16 19:06:03.802911 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.802852    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec314ed4c9c8de17440a9ff15a8076f700c8680e81d14b7680bf7fd4ee2108a"} err="failed to get container status \"4ec314ed4c9c8de17440a9ff15a8076f700c8680e81d14b7680bf7fd4ee2108a\": rpc error: code = NotFound desc = could not find container \"4ec314ed4c9c8de17440a9ff15a8076f700c8680e81d14b7680bf7fd4ee2108a\": container with ID starting with 4ec314ed4c9c8de17440a9ff15a8076f700c8680e81d14b7680bf7fd4ee2108a not found: ID does not exist"
Sep 16 19:06:03.802911 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.802871    2582 scope.go:117] "RemoveContainer" containerID="57ff5d60fa773c82bde5d7e26f5458520a53413c6c46e11c2afa4cc851068bc7"
Sep 16 19:06:03.803154 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:06:03.803135    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ff5d60fa773c82bde5d7e26f5458520a53413c6c46e11c2afa4cc851068bc7\": container with ID starting with 57ff5d60fa773c82bde5d7e26f5458520a53413c6c46e11c2afa4cc851068bc7 not found: ID does not exist" containerID="57ff5d60fa773c82bde5d7e26f5458520a53413c6c46e11c2afa4cc851068bc7"
Sep 16 19:06:03.803207 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.803157    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ff5d60fa773c82bde5d7e26f5458520a53413c6c46e11c2afa4cc851068bc7"} err="failed to get container status \"57ff5d60fa773c82bde5d7e26f5458520a53413c6c46e11c2afa4cc851068bc7\": rpc error: code = NotFound desc = could not find container \"57ff5d60fa773c82bde5d7e26f5458520a53413c6c46e11c2afa4cc851068bc7\": container with ID starting with 57ff5d60fa773c82bde5d7e26f5458520a53413c6c46e11c2afa4cc851068bc7 not found: ID does not exist"
Sep 16 19:06:03.808282 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.808257    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"]
Sep 16 19:06:03.813726 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.813692    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-86c6b664b7-7cgfk"]
Sep 16 19:06:03.972809 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:03.972768    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h" podUID="8adc7e9f-6094-4860-9715-deeed44f78dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:04.703754 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:04.703713    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" path="/var/lib/kubelet/pods/d12c8784-dbe5-4995-8b0f-418b6b8ffb7c/volumes"
Sep 16 19:06:07.245090 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:07.245057    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:13.973115 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:13.973068    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h" podUID="8adc7e9f-6094-4860-9715-deeed44f78dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:13.973563 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:13.973171    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"
Sep 16 19:06:23.973246 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:23.973197    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h" podUID="8adc7e9f-6094-4860-9715-deeed44f78dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:31.662371 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:31.662311    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:31.727524 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:31.727473    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:32.835644 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:32.835614    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"
Sep 16 19:06:32.899794 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:32.899720    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dfxh\" (UniqueName: \"kubernetes.io/projected/8adc7e9f-6094-4860-9715-deeed44f78dd-kube-api-access-5dfxh\") pod \"8adc7e9f-6094-4860-9715-deeed44f78dd\" (UID: \"8adc7e9f-6094-4860-9715-deeed44f78dd\") "
Sep 16 19:06:32.899943 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:32.899829    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8adc7e9f-6094-4860-9715-deeed44f78dd-tls-certs\") pod \"8adc7e9f-6094-4860-9715-deeed44f78dd\" (UID: \"8adc7e9f-6094-4860-9715-deeed44f78dd\") "
Sep 16 19:06:32.901937 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:32.901913    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8adc7e9f-6094-4860-9715-deeed44f78dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8adc7e9f-6094-4860-9715-deeed44f78dd" (UID: "8adc7e9f-6094-4860-9715-deeed44f78dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:32.902039 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:32.901949    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8adc7e9f-6094-4860-9715-deeed44f78dd-kube-api-access-5dfxh" (OuterVolumeSpecName: "kube-api-access-5dfxh") pod "8adc7e9f-6094-4860-9715-deeed44f78dd" (UID: "8adc7e9f-6094-4860-9715-deeed44f78dd"). InnerVolumeSpecName "kube-api-access-5dfxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:32.911531 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:32.911504    2582 generic.go:356] "Generic (PLEG): container finished" podID="8adc7e9f-6094-4860-9715-deeed44f78dd" containerID="a03a6e8ab877d2c6bdd323b45fb56cd8f9a33ea5bb9b4522967ef498c3f1a5f3" exitCode=0
Sep 16 19:06:32.911658 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:32.911550    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"
Sep 16 19:06:32.911658 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:32.911553    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h" event={"ID":"8adc7e9f-6094-4860-9715-deeed44f78dd","Type":"ContainerDied","Data":"a03a6e8ab877d2c6bdd323b45fb56cd8f9a33ea5bb9b4522967ef498c3f1a5f3"}
Sep 16 19:06:32.911658 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:32.911635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h" event={"ID":"8adc7e9f-6094-4860-9715-deeed44f78dd","Type":"ContainerDied","Data":"f78ad11df7b1b4eb5e9056a39c5f4e5f9d4850f148bda8240959cb9a22101b99"}
Sep 16 19:06:32.911888 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:32.911665    2582 scope.go:117] "RemoveContainer" containerID="a03a6e8ab877d2c6bdd323b45fb56cd8f9a33ea5bb9b4522967ef498c3f1a5f3"
Sep 16 19:06:32.920412 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:32.920395    2582 scope.go:117] "RemoveContainer" containerID="a03a6e8ab877d2c6bdd323b45fb56cd8f9a33ea5bb9b4522967ef498c3f1a5f3"
Sep 16 19:06:32.920691 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:06:32.920673    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03a6e8ab877d2c6bdd323b45fb56cd8f9a33ea5bb9b4522967ef498c3f1a5f3\": container with ID starting with a03a6e8ab877d2c6bdd323b45fb56cd8f9a33ea5bb9b4522967ef498c3f1a5f3 not found: ID does not exist" containerID="a03a6e8ab877d2c6bdd323b45fb56cd8f9a33ea5bb9b4522967ef498c3f1a5f3"
Sep 16 19:06:32.920763 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:32.920695    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03a6e8ab877d2c6bdd323b45fb56cd8f9a33ea5bb9b4522967ef498c3f1a5f3"} err="failed to get container status \"a03a6e8ab877d2c6bdd323b45fb56cd8f9a33ea5bb9b4522967ef498c3f1a5f3\": rpc error: code = NotFound desc = could not find container \"a03a6e8ab877d2c6bdd323b45fb56cd8f9a33ea5bb9b4522967ef498c3f1a5f3\": container with ID starting with a03a6e8ab877d2c6bdd323b45fb56cd8f9a33ea5bb9b4522967ef498c3f1a5f3 not found: ID does not exist"
Sep 16 19:06:32.933882 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:32.933857    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"]
Sep 16 19:06:32.940433 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:32.940416    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-c899h"]
Sep 16 19:06:33.000386 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:33.000370    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8adc7e9f-6094-4860-9715-deeed44f78dd-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:06:33.000492 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:33.000389    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dfxh\" (UniqueName: \"kubernetes.io/projected/8adc7e9f-6094-4860-9715-deeed44f78dd-kube-api-access-5dfxh\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:06:34.703183 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:34.703153    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8adc7e9f-6094-4860-9715-deeed44f78dd" path="/var/lib/kubelet/pods/8adc7e9f-6094-4860-9715-deeed44f78dd/volumes"
Sep 16 19:06:37.664442 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:37.664412    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:37.760349 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:37.760320    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:43.760782 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.760680    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:43.929029 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.928996    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"]
Sep 16 19:06:43.929344 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.929331    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="storage-initializer"
Sep 16 19:06:43.929399 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.929346    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="storage-initializer"
Sep 16 19:06:43.929399 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.929358    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main"
Sep 16 19:06:43.929399 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.929363    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main"
Sep 16 19:06:43.929399 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.929375    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8adc7e9f-6094-4860-9715-deeed44f78dd" containerName="main"
Sep 16 19:06:43.929399 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.929381    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8adc7e9f-6094-4860-9715-deeed44f78dd" containerName="main"
Sep 16 19:06:43.929594 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.929431    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12c8784-dbe5-4995-8b0f-418b6b8ffb7c" containerName="main"
Sep 16 19:06:43.929594 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.929439    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8adc7e9f-6094-4860-9715-deeed44f78dd" containerName="main"
Sep 16 19:06:43.933768 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.933746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:43.936871 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.936847    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 16 19:06:43.937033 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.936917    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:06:43.937033 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.936976    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 19:06:43.937163 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.937075    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 19:06:43.937163 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.937152    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 19:06:43.937305 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.937286    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-gj95q\""
Sep 16 19:06:43.946496 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:43.946468    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"]
Sep 16 19:06:44.095390 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.095344    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-dshm\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.095390 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.095398    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6286a278-c312-4d1d-ab4d-1edfec9b667f-tls-certs\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.095685 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.095461    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.095685 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.095542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-home\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.095685 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.095607    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-tmp-dir\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.095685 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.095626    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-model-cache\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.095685 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.095672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6286a278-c312-4d1d-ab4d-1edfec9b667f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.095903 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.095699    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjvdn\" (UniqueName: \"kubernetes.io/projected/6286a278-c312-4d1d-ab4d-1edfec9b667f-kube-api-access-zjvdn\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.127984 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.127947    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"]
Sep 16 19:06:44.131402 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.131376    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"
Sep 16 19:06:44.134280 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.134253    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-m8s64\""
Sep 16 19:06:44.144408 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.144378    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"]
Sep 16 19:06:44.196515 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.196474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-home\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.196743 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.196529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-tmp-dir\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.196743 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.196551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-model-cache\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.196743 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.196605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6286a278-c312-4d1d-ab4d-1edfec9b667f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.196743 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.196644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjvdn\" (UniqueName: \"kubernetes.io/projected/6286a278-c312-4d1d-ab4d-1edfec9b667f-kube-api-access-zjvdn\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.196743 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.196684    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-dshm\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.196743 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.196728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6286a278-c312-4d1d-ab4d-1edfec9b667f-tls-certs\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.197069 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.196754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.197069 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.197029    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-tmp-dir\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.197179 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.197069    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-home\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.197179 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.197108    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-model-cache\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.197179 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.197133    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.197475 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.197450    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6286a278-c312-4d1d-ab4d-1edfec9b667f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.199239 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.199209    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-dshm\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.199564 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.199543    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6286a278-c312-4d1d-ab4d-1edfec9b667f-tls-certs\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.205711 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.205681    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjvdn\" (UniqueName: \"kubernetes.io/projected/6286a278-c312-4d1d-ab4d-1edfec9b667f-kube-api-access-zjvdn\") pod \"auth-invalid-token-test-kserve-755ddb54dc-4zmq4\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.243946 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.243903    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:44.298073 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.298044    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s57dj\" (UniqueName: \"kubernetes.io/projected/8f3934c1-62a8-4542-9481-f8cc26b16cad-kube-api-access-s57dj\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d\" (UID: \"8f3934c1-62a8-4542-9481-f8cc26b16cad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"
Sep 16 19:06:44.298292 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.298094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f3934c1-62a8-4542-9481-f8cc26b16cad-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d\" (UID: \"8f3934c1-62a8-4542-9481-f8cc26b16cad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"
Sep 16 19:06:44.398881 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.398766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f3934c1-62a8-4542-9481-f8cc26b16cad-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d\" (UID: \"8f3934c1-62a8-4542-9481-f8cc26b16cad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"
Sep 16 19:06:44.399069 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.398927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s57dj\" (UniqueName: \"kubernetes.io/projected/8f3934c1-62a8-4542-9481-f8cc26b16cad-kube-api-access-s57dj\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d\" (UID: \"8f3934c1-62a8-4542-9481-f8cc26b16cad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"
Sep 16 19:06:44.401864 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.401824    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f3934c1-62a8-4542-9481-f8cc26b16cad-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d\" (UID: \"8f3934c1-62a8-4542-9481-f8cc26b16cad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"
Sep 16 19:06:44.408495 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.408449    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s57dj\" (UniqueName: \"kubernetes.io/projected/8f3934c1-62a8-4542-9481-f8cc26b16cad-kube-api-access-s57dj\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d\" (UID: \"8f3934c1-62a8-4542-9481-f8cc26b16cad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"
Sep 16 19:06:44.443357 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.443314    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"
Sep 16 19:06:44.527157 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.527104    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"]
Sep 16 19:06:44.532864 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:06:44.532818    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6286a278_c312_4d1d_ab4d_1edfec9b667f.slice/crio-c86e0fd4aaf68bf7a50fe4a4c4f94b6ed8989a1865984e1e23acca63f83e8f8e WatchSource:0}: Error finding container c86e0fd4aaf68bf7a50fe4a4c4f94b6ed8989a1865984e1e23acca63f83e8f8e: Status 404 returned error can't find the container with id c86e0fd4aaf68bf7a50fe4a4c4f94b6ed8989a1865984e1e23acca63f83e8f8e
Sep 16 19:06:44.727785 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.727754    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"]
Sep 16 19:06:44.730143 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:06:44.730107    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3934c1_62a8_4542_9481_f8cc26b16cad.slice/crio-af526fb8e1fc13cee36cf5af8acfc2e38ad807f91f7bd77d93c244b08a08e30e WatchSource:0}: Error finding container af526fb8e1fc13cee36cf5af8acfc2e38ad807f91f7bd77d93c244b08a08e30e: Status 404 returned error can't find the container with id af526fb8e1fc13cee36cf5af8acfc2e38ad807f91f7bd77d93c244b08a08e30e
Sep 16 19:06:44.968242 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.968204    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d" event={"ID":"8f3934c1-62a8-4542-9481-f8cc26b16cad","Type":"ContainerStarted","Data":"1db20175f834fb011930f6cab82816133c7f82e6497bbb906fb8d5edf4475192"}
Sep 16 19:06:44.968724 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.968267    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d" event={"ID":"8f3934c1-62a8-4542-9481-f8cc26b16cad","Type":"ContainerStarted","Data":"af526fb8e1fc13cee36cf5af8acfc2e38ad807f91f7bd77d93c244b08a08e30e"}
Sep 16 19:06:44.968724 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.968299    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"
Sep 16 19:06:44.970010 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.969983    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" event={"ID":"6286a278-c312-4d1d-ab4d-1edfec9b667f","Type":"ContainerStarted","Data":"d610b18f0176b29f17d87cd9f4ab934ec63ccd177c8a53d14fc816e7d09f3b3f"}
Sep 16 19:06:44.970165 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.970026    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" event={"ID":"6286a278-c312-4d1d-ab4d-1edfec9b667f","Type":"ContainerStarted","Data":"c86e0fd4aaf68bf7a50fe4a4c4f94b6ed8989a1865984e1e23acca63f83e8f8e"}
Sep 16 19:06:44.990504 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:44.990395    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d" podStartSLOduration=0.990378297 podStartE2EDuration="990.378297ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:44.98891613 +0000 UTC m=+1262.819524676" watchObservedRunningTime="2026-09-16 19:06:44.990378297 +0000 UTC m=+1262.820986842"
Sep 16 19:06:45.485962 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:45.485926    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:47.985708 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:47.985672    2582 generic.go:356] "Generic (PLEG): container finished" podID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerID="d610b18f0176b29f17d87cd9f4ab934ec63ccd177c8a53d14fc816e7d09f3b3f" exitCode=0
Sep 16 19:06:47.986118 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:47.985747    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" event={"ID":"6286a278-c312-4d1d-ab4d-1edfec9b667f","Type":"ContainerDied","Data":"d610b18f0176b29f17d87cd9f4ab934ec63ccd177c8a53d14fc816e7d09f3b3f"}
Sep 16 19:06:48.995759 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:48.995701    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" event={"ID":"6286a278-c312-4d1d-ab4d-1edfec9b667f","Type":"ContainerStarted","Data":"8588732facdb481bb2077f45a8da8f2da1215febae365d32f224928e52f5fae8"}
Sep 16 19:06:49.022396 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:49.022312    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podStartSLOduration=3.629740485 podStartE2EDuration="6.022276833s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.392536348s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:49.018966732 +0000 UTC m=+1266.849575283" watchObservedRunningTime="2026-09-16 19:06:49.022276833 +0000 UTC m=+1266.852885385"
Sep 16 19:06:49.244681 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:49.244631    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:49.287203 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:49.287161    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:49.356656 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:49.356616    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:52.596548 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:52.596492    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:54.244756 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:54.244718    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:06:54.246452 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:06:54.246414    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 19:07:04.245418 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:04.245352    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 19:07:14.245065 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:14.245013    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 19:07:15.977371 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:15.977335    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"
Sep 16 19:07:21.635108 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:21.635063    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:21.778928 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:21.778891    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:24.245419 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:24.245360    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 19:07:27.620046 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:27.619996    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:28.214844 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:28.214794    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:28.281243 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:28.281189    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:30.841769 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:30.841726    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:31.043530 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.043497    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"]
Sep 16 19:07:31.047515 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.047495    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"
Sep 16 19:07:31.050718 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.050690    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 19:07:31.052080 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.052048    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 19:07:31.052266 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.052138    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-nhj5b\""
Sep 16 19:07:31.052266 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.052190    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 19:07:31.052370 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.052271    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-hpkpc\""
Sep 16 19:07:31.058051 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.058025    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"]
Sep 16 19:07:31.097986 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.097892    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:31.162324 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.162286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd882ea6-16d9-4e69-a81a-4b604b0214c1-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld\" (UID: \"bd882ea6-16d9-4e69-a81a-4b604b0214c1\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"
Sep 16 19:07:31.162324 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.162328    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fszmb\" (UniqueName: \"kubernetes.io/projected/bd882ea6-16d9-4e69-a81a-4b604b0214c1-kube-api-access-fszmb\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld\" (UID: \"bd882ea6-16d9-4e69-a81a-4b604b0214c1\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"
Sep 16 19:07:31.262943 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.262884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd882ea6-16d9-4e69-a81a-4b604b0214c1-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld\" (UID: \"bd882ea6-16d9-4e69-a81a-4b604b0214c1\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"
Sep 16 19:07:31.262943 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.262947    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fszmb\" (UniqueName: \"kubernetes.io/projected/bd882ea6-16d9-4e69-a81a-4b604b0214c1-kube-api-access-fszmb\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld\" (UID: \"bd882ea6-16d9-4e69-a81a-4b604b0214c1\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"
Sep 16 19:07:31.266324 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.266296    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd882ea6-16d9-4e69-a81a-4b604b0214c1-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld\" (UID: \"bd882ea6-16d9-4e69-a81a-4b604b0214c1\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"
Sep 16 19:07:31.272598 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.272527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fszmb\" (UniqueName: \"kubernetes.io/projected/bd882ea6-16d9-4e69-a81a-4b604b0214c1-kube-api-access-fszmb\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld\" (UID: \"bd882ea6-16d9-4e69-a81a-4b604b0214c1\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"
Sep 16 19:07:31.360610 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.360462    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"
Sep 16 19:07:31.645972 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:31.645933    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"]
Sep 16 19:07:32.208518 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:32.208464    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld" event={"ID":"bd882ea6-16d9-4e69-a81a-4b604b0214c1","Type":"ContainerStarted","Data":"5dd02cb05b0f602f1b7b499c58bea430924a09ad040d3d4cf8b2d27463fbaa72"}
Sep 16 19:07:32.209041 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:32.208539    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld" event={"ID":"bd882ea6-16d9-4e69-a81a-4b604b0214c1","Type":"ContainerStarted","Data":"c8648fcc88b2e5e71c45fc9fec93214eb7ab1cc9444f92d31d3e78b0bb909b80"}
Sep 16 19:07:32.209041 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:32.208715    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"
Sep 16 19:07:32.227073 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:32.227005    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld" podStartSLOduration=1.226983493 podStartE2EDuration="1.226983493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:32.224725906 +0000 UTC m=+1310.055334450" watchObservedRunningTime="2026-09-16 19:07:32.226983493 +0000 UTC m=+1310.057592037"
Sep 16 19:07:34.049137 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:34.049087    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:34.150210 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:34.150161    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:34.245060 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:34.245009    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 19:07:43.090317 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:43.090270    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:43.194768 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:43.194726    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:43.290733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:43.290692    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:44.245430 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:44.245375    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 19:07:54.244986 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:07:54.244934    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 19:08:03.216471 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:03.216359    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld" podUID="bd882ea6-16d9-4e69-a81a-4b604b0214c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:04.245319 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:04.245264    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 19:08:05.233157 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:05.233099    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"]
Sep 16 19:08:05.235399 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:05.235358    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld" podUID="bd882ea6-16d9-4e69-a81a-4b604b0214c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:05.237519 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:05.237486    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:05.315412 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:05.315377    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:07.933189 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:07.933134    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:14.244459 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:14.244412    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 19:08:15.235592 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:15.235532    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld" podUID="bd882ea6-16d9-4e69-a81a-4b604b0214c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:19.476838 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:19.476799    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:20.257083 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:20.257026    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld" podUID="bd882ea6-16d9-4e69-a81a-4b604b0214c1" containerName="main" containerID="cri-o://5dd02cb05b0f602f1b7b499c58bea430924a09ad040d3d4cf8b2d27463fbaa72" gracePeriod=45
Sep 16 19:08:24.244952 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:24.244894    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 19:08:25.235946 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:25.235890    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld" podUID="bd882ea6-16d9-4e69-a81a-4b604b0214c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:34.244888 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:34.244836    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 19:08:35.235294 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:35.235244    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld" podUID="bd882ea6-16d9-4e69-a81a-4b604b0214c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:44.245349 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:44.245288    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 19:08:45.236003 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:45.235955    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld" podUID="bd882ea6-16d9-4e69-a81a-4b604b0214c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:50.506911 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.506876    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"
Sep 16 19:08:50.561402 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.561368    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fszmb\" (UniqueName: \"kubernetes.io/projected/bd882ea6-16d9-4e69-a81a-4b604b0214c1-kube-api-access-fszmb\") pod \"bd882ea6-16d9-4e69-a81a-4b604b0214c1\" (UID: \"bd882ea6-16d9-4e69-a81a-4b604b0214c1\") "
Sep 16 19:08:50.561626 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.561416    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd882ea6-16d9-4e69-a81a-4b604b0214c1-tls-certs\") pod \"bd882ea6-16d9-4e69-a81a-4b604b0214c1\" (UID: \"bd882ea6-16d9-4e69-a81a-4b604b0214c1\") "
Sep 16 19:08:50.564035 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.564004    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd882ea6-16d9-4e69-a81a-4b604b0214c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd882ea6-16d9-4e69-a81a-4b604b0214c1" (UID: "bd882ea6-16d9-4e69-a81a-4b604b0214c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:50.564179 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.564034    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd882ea6-16d9-4e69-a81a-4b604b0214c1-kube-api-access-fszmb" (OuterVolumeSpecName: "kube-api-access-fszmb") pod "bd882ea6-16d9-4e69-a81a-4b604b0214c1" (UID: "bd882ea6-16d9-4e69-a81a-4b604b0214c1"). InnerVolumeSpecName "kube-api-access-fszmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:50.632054 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.632010    2582 generic.go:356] "Generic (PLEG): container finished" podID="bd882ea6-16d9-4e69-a81a-4b604b0214c1" containerID="5dd02cb05b0f602f1b7b499c58bea430924a09ad040d3d4cf8b2d27463fbaa72" exitCode=0
Sep 16 19:08:50.632250 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.632061    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld" event={"ID":"bd882ea6-16d9-4e69-a81a-4b604b0214c1","Type":"ContainerDied","Data":"5dd02cb05b0f602f1b7b499c58bea430924a09ad040d3d4cf8b2d27463fbaa72"}
Sep 16 19:08:50.632250 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.632085    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"
Sep 16 19:08:50.632250 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.632156    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld" event={"ID":"bd882ea6-16d9-4e69-a81a-4b604b0214c1","Type":"ContainerDied","Data":"c8648fcc88b2e5e71c45fc9fec93214eb7ab1cc9444f92d31d3e78b0bb909b80"}
Sep 16 19:08:50.632250 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.632189    2582 scope.go:117] "RemoveContainer" containerID="5dd02cb05b0f602f1b7b499c58bea430924a09ad040d3d4cf8b2d27463fbaa72"
Sep 16 19:08:50.642380 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.642356    2582 scope.go:117] "RemoveContainer" containerID="5dd02cb05b0f602f1b7b499c58bea430924a09ad040d3d4cf8b2d27463fbaa72"
Sep 16 19:08:50.642754 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:08:50.642730    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd02cb05b0f602f1b7b499c58bea430924a09ad040d3d4cf8b2d27463fbaa72\": container with ID starting with 5dd02cb05b0f602f1b7b499c58bea430924a09ad040d3d4cf8b2d27463fbaa72 not found: ID does not exist" containerID="5dd02cb05b0f602f1b7b499c58bea430924a09ad040d3d4cf8b2d27463fbaa72"
Sep 16 19:08:50.642860 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.642761    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd02cb05b0f602f1b7b499c58bea430924a09ad040d3d4cf8b2d27463fbaa72"} err="failed to get container status \"5dd02cb05b0f602f1b7b499c58bea430924a09ad040d3d4cf8b2d27463fbaa72\": rpc error: code = NotFound desc = could not find container \"5dd02cb05b0f602f1b7b499c58bea430924a09ad040d3d4cf8b2d27463fbaa72\": container with ID starting with 5dd02cb05b0f602f1b7b499c58bea430924a09ad040d3d4cf8b2d27463fbaa72 not found: ID does not exist"
Sep 16 19:08:50.657509 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.657478    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"]
Sep 16 19:08:50.660441 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.660406    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527jld"]
Sep 16 19:08:50.662186 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.662167    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fszmb\" (UniqueName: \"kubernetes.io/projected/bd882ea6-16d9-4e69-a81a-4b604b0214c1-kube-api-access-fszmb\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:08:50.662240 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.662189    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd882ea6-16d9-4e69-a81a-4b604b0214c1-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:08:50.705483 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:50.705437    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd882ea6-16d9-4e69-a81a-4b604b0214c1" path="/var/lib/kubelet/pods/bd882ea6-16d9-4e69-a81a-4b604b0214c1/volumes"
Sep 16 19:08:54.255600 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:54.255536    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:08:54.263683 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:54.263654    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:08:55.256207 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:55.256164    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:55.364533 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:08:55.364495    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:01.286213 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:01.286181    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:06.932565 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:06.932529    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:07.032261 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:07.032203    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:11.700731 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:11.700693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:11.896630 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:11.896597    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:23.419164 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:23.419127    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:24.945874 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:24.945840    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"]
Sep 16 19:09:24.952722 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:24.952686    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"]
Sep 16 19:09:24.955802 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:24.955771    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:25.024851 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:25.024817    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:25.976878 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:25.976840    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d" podUID="8f3934c1-62a8-4542-9481-f8cc26b16cad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:35.976768 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:35.976678    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d" podUID="8f3934c1-62a8-4542-9481-f8cc26b16cad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:39.968021 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:39.967985    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d" podUID="8f3934c1-62a8-4542-9481-f8cc26b16cad" containerName="main" containerID="cri-o://1db20175f834fb011930f6cab82816133c7f82e6497bbb906fb8d5edf4475192" gracePeriod=45
Sep 16 19:09:39.976685 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:39.976655    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main" containerID="cri-o://8588732facdb481bb2077f45a8da8f2da1215febae365d32f224928e52f5fae8" gracePeriod=45
Sep 16 19:09:40.256937 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.256851    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Sep 16 19:09:40.628529 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.628505    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:09:40.723054 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.723019    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6286a278-c312-4d1d-ab4d-1edfec9b667f-cabundle-cert\") pod \"6286a278-c312-4d1d-ab4d-1edfec9b667f\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") "
Sep 16 19:09:40.723228 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.723087    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-tmp-dir\") pod \"6286a278-c312-4d1d-ab4d-1edfec9b667f\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") "
Sep 16 19:09:40.723228 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.723150    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-model-cache\") pod \"6286a278-c312-4d1d-ab4d-1edfec9b667f\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") "
Sep 16 19:09:40.723228 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.723181    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6286a278-c312-4d1d-ab4d-1edfec9b667f-tls-certs\") pod \"6286a278-c312-4d1d-ab4d-1edfec9b667f\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") "
Sep 16 19:09:40.723228 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.723198    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-home\") pod \"6286a278-c312-4d1d-ab4d-1edfec9b667f\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") "
Sep 16 19:09:40.723492 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.723335    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-dshm\") pod \"6286a278-c312-4d1d-ab4d-1edfec9b667f\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") "
Sep 16 19:09:40.723492 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.723380    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjvdn\" (UniqueName: \"kubernetes.io/projected/6286a278-c312-4d1d-ab4d-1edfec9b667f-kube-api-access-zjvdn\") pod \"6286a278-c312-4d1d-ab4d-1edfec9b667f\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") "
Sep 16 19:09:40.723492 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.723440    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-kserve-provision-location\") pod \"6286a278-c312-4d1d-ab4d-1edfec9b667f\" (UID: \"6286a278-c312-4d1d-ab4d-1edfec9b667f\") "
Sep 16 19:09:40.723741 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.723404    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6286a278-c312-4d1d-ab4d-1edfec9b667f" (UID: "6286a278-c312-4d1d-ab4d-1edfec9b667f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:40.723809 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.723528    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6286a278-c312-4d1d-ab4d-1edfec9b667f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6286a278-c312-4d1d-ab4d-1edfec9b667f" (UID: "6286a278-c312-4d1d-ab4d-1edfec9b667f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:40.723954 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.723935    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-home" (OuterVolumeSpecName: "home") pod "6286a278-c312-4d1d-ab4d-1edfec9b667f" (UID: "6286a278-c312-4d1d-ab4d-1edfec9b667f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:40.726194 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.726167    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-dshm" (OuterVolumeSpecName: "dshm") pod "6286a278-c312-4d1d-ab4d-1edfec9b667f" (UID: "6286a278-c312-4d1d-ab4d-1edfec9b667f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:40.726194 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.726178    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6286a278-c312-4d1d-ab4d-1edfec9b667f-kube-api-access-zjvdn" (OuterVolumeSpecName: "kube-api-access-zjvdn") pod "6286a278-c312-4d1d-ab4d-1edfec9b667f" (UID: "6286a278-c312-4d1d-ab4d-1edfec9b667f"). InnerVolumeSpecName "kube-api-access-zjvdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:40.726381 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.726352    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6286a278-c312-4d1d-ab4d-1edfec9b667f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6286a278-c312-4d1d-ab4d-1edfec9b667f" (UID: "6286a278-c312-4d1d-ab4d-1edfec9b667f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:40.735513 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.735487    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6286a278-c312-4d1d-ab4d-1edfec9b667f" (UID: "6286a278-c312-4d1d-ab4d-1edfec9b667f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:40.746108 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.746065    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6286a278-c312-4d1d-ab4d-1edfec9b667f" (UID: "6286a278-c312-4d1d-ab4d-1edfec9b667f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:40.825197 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.825113    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:09:40.825197 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.825147    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6286a278-c312-4d1d-ab4d-1edfec9b667f-cabundle-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:09:40.825197 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.825158    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:09:40.825197 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.825165    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:09:40.825197 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.825173    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6286a278-c312-4d1d-ab4d-1edfec9b667f-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:09:40.825197 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.825180    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:09:40.825197 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.825187    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6286a278-c312-4d1d-ab4d-1edfec9b667f-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:09:40.825197 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.825195    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjvdn\" (UniqueName: \"kubernetes.io/projected/6286a278-c312-4d1d-ab4d-1edfec9b667f-kube-api-access-zjvdn\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:09:40.880487 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.880454    2582 generic.go:356] "Generic (PLEG): container finished" podID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerID="8588732facdb481bb2077f45a8da8f2da1215febae365d32f224928e52f5fae8" exitCode=0
Sep 16 19:09:40.880642 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.880516    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" event={"ID":"6286a278-c312-4d1d-ab4d-1edfec9b667f","Type":"ContainerDied","Data":"8588732facdb481bb2077f45a8da8f2da1215febae365d32f224928e52f5fae8"}
Sep 16 19:09:40.880642 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.880565    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4" event={"ID":"6286a278-c312-4d1d-ab4d-1edfec9b667f","Type":"ContainerDied","Data":"c86e0fd4aaf68bf7a50fe4a4c4f94b6ed8989a1865984e1e23acca63f83e8f8e"}
Sep 16 19:09:40.880642 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.880525    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"
Sep 16 19:09:40.880818 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.880611    2582 scope.go:117] "RemoveContainer" containerID="8588732facdb481bb2077f45a8da8f2da1215febae365d32f224928e52f5fae8"
Sep 16 19:09:40.891447 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.891419    2582 scope.go:117] "RemoveContainer" containerID="d610b18f0176b29f17d87cd9f4ab934ec63ccd177c8a53d14fc816e7d09f3b3f"
Sep 16 19:09:40.899449 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.899433    2582 scope.go:117] "RemoveContainer" containerID="8588732facdb481bb2077f45a8da8f2da1215febae365d32f224928e52f5fae8"
Sep 16 19:09:40.899755 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:09:40.899735    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8588732facdb481bb2077f45a8da8f2da1215febae365d32f224928e52f5fae8\": container with ID starting with 8588732facdb481bb2077f45a8da8f2da1215febae365d32f224928e52f5fae8 not found: ID does not exist" containerID="8588732facdb481bb2077f45a8da8f2da1215febae365d32f224928e52f5fae8"
Sep 16 19:09:40.899845 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.899762    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8588732facdb481bb2077f45a8da8f2da1215febae365d32f224928e52f5fae8"} err="failed to get container status \"8588732facdb481bb2077f45a8da8f2da1215febae365d32f224928e52f5fae8\": rpc error: code = NotFound desc = could not find container \"8588732facdb481bb2077f45a8da8f2da1215febae365d32f224928e52f5fae8\": container with ID starting with 8588732facdb481bb2077f45a8da8f2da1215febae365d32f224928e52f5fae8 not found: ID does not exist"
Sep 16 19:09:40.899845 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.899780    2582 scope.go:117] "RemoveContainer" containerID="d610b18f0176b29f17d87cd9f4ab934ec63ccd177c8a53d14fc816e7d09f3b3f"
Sep 16 19:09:40.900026 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:09:40.900013    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d610b18f0176b29f17d87cd9f4ab934ec63ccd177c8a53d14fc816e7d09f3b3f\": container with ID starting with d610b18f0176b29f17d87cd9f4ab934ec63ccd177c8a53d14fc816e7d09f3b3f not found: ID does not exist" containerID="d610b18f0176b29f17d87cd9f4ab934ec63ccd177c8a53d14fc816e7d09f3b3f"
Sep 16 19:09:40.900091 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.900035    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d610b18f0176b29f17d87cd9f4ab934ec63ccd177c8a53d14fc816e7d09f3b3f"} err="failed to get container status \"d610b18f0176b29f17d87cd9f4ab934ec63ccd177c8a53d14fc816e7d09f3b3f\": rpc error: code = NotFound desc = could not find container \"d610b18f0176b29f17d87cd9f4ab934ec63ccd177c8a53d14fc816e7d09f3b3f\": container with ID starting with d610b18f0176b29f17d87cd9f4ab934ec63ccd177c8a53d14fc816e7d09f3b3f not found: ID does not exist"
Sep 16 19:09:40.906730 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.906710    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"]
Sep 16 19:09:40.910277 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:40.910259    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-755ddb54dc-4zmq4"]
Sep 16 19:09:42.704057 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:42.704012    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" path="/var/lib/kubelet/pods/6286a278-c312-4d1d-ab4d-1edfec9b667f/volumes"
Sep 16 19:09:45.764746 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:45.764718    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:45.885601 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:45.885541    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:45.976714 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:45.976676    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d" podUID="8f3934c1-62a8-4542-9481-f8cc26b16cad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:45.976897 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:45.976794    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"
Sep 16 19:09:55.976567 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:09:55.976528    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d" podUID="8f3934c1-62a8-4542-9481-f8cc26b16cad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:10:05.977206 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:05.977159    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d" podUID="8f3934c1-62a8-4542-9481-f8cc26b16cad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:10:10.220888 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:10.220858    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"
Sep 16 19:10:10.271115 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:10.271084    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s57dj\" (UniqueName: \"kubernetes.io/projected/8f3934c1-62a8-4542-9481-f8cc26b16cad-kube-api-access-s57dj\") pod \"8f3934c1-62a8-4542-9481-f8cc26b16cad\" (UID: \"8f3934c1-62a8-4542-9481-f8cc26b16cad\") "
Sep 16 19:10:10.271327 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:10.271134    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f3934c1-62a8-4542-9481-f8cc26b16cad-tls-certs\") pod \"8f3934c1-62a8-4542-9481-f8cc26b16cad\" (UID: \"8f3934c1-62a8-4542-9481-f8cc26b16cad\") "
Sep 16 19:10:10.273471 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:10.273432    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3934c1-62a8-4542-9481-f8cc26b16cad-kube-api-access-s57dj" (OuterVolumeSpecName: "kube-api-access-s57dj") pod "8f3934c1-62a8-4542-9481-f8cc26b16cad" (UID: "8f3934c1-62a8-4542-9481-f8cc26b16cad"). InnerVolumeSpecName "kube-api-access-s57dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:10.273471 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:10.273456    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f3934c1-62a8-4542-9481-f8cc26b16cad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f3934c1-62a8-4542-9481-f8cc26b16cad" (UID: "8f3934c1-62a8-4542-9481-f8cc26b16cad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:10.371778 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:10.371733    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f3934c1-62a8-4542-9481-f8cc26b16cad-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:10:10.371778 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:10.371771    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s57dj\" (UniqueName: \"kubernetes.io/projected/8f3934c1-62a8-4542-9481-f8cc26b16cad-kube-api-access-s57dj\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:10:11.019547 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:11.019510    2582 generic.go:356] "Generic (PLEG): container finished" podID="8f3934c1-62a8-4542-9481-f8cc26b16cad" containerID="1db20175f834fb011930f6cab82816133c7f82e6497bbb906fb8d5edf4475192" exitCode=0
Sep 16 19:10:11.019773 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:11.019568    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"
Sep 16 19:10:11.019773 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:11.019607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d" event={"ID":"8f3934c1-62a8-4542-9481-f8cc26b16cad","Type":"ContainerDied","Data":"1db20175f834fb011930f6cab82816133c7f82e6497bbb906fb8d5edf4475192"}
Sep 16 19:10:11.019773 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:11.019662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d" event={"ID":"8f3934c1-62a8-4542-9481-f8cc26b16cad","Type":"ContainerDied","Data":"af526fb8e1fc13cee36cf5af8acfc2e38ad807f91f7bd77d93c244b08a08e30e"}
Sep 16 19:10:11.019773 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:11.019686    2582 scope.go:117] "RemoveContainer" containerID="1db20175f834fb011930f6cab82816133c7f82e6497bbb906fb8d5edf4475192"
Sep 16 19:10:11.028431 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:11.028412    2582 scope.go:117] "RemoveContainer" containerID="1db20175f834fb011930f6cab82816133c7f82e6497bbb906fb8d5edf4475192"
Sep 16 19:10:11.028808 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:10:11.028776    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db20175f834fb011930f6cab82816133c7f82e6497bbb906fb8d5edf4475192\": container with ID starting with 1db20175f834fb011930f6cab82816133c7f82e6497bbb906fb8d5edf4475192 not found: ID does not exist" containerID="1db20175f834fb011930f6cab82816133c7f82e6497bbb906fb8d5edf4475192"
Sep 16 19:10:11.028861 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:11.028819    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db20175f834fb011930f6cab82816133c7f82e6497bbb906fb8d5edf4475192"} err="failed to get container status \"1db20175f834fb011930f6cab82816133c7f82e6497bbb906fb8d5edf4475192\": rpc error: code = NotFound desc = could not find container \"1db20175f834fb011930f6cab82816133c7f82e6497bbb906fb8d5edf4475192\": container with ID starting with 1db20175f834fb011930f6cab82816133c7f82e6497bbb906fb8d5edf4475192 not found: ID does not exist"
Sep 16 19:10:11.039736 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:11.039708    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"]
Sep 16 19:10:11.044416 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:11.044385    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cdh52d"]
Sep 16 19:10:11.474442 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:11.474344    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:11.573347 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:11.573310    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:12.705076 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:12.705036    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f3934c1-62a8-4542-9481-f8cc26b16cad" path="/var/lib/kubelet/pods/8f3934c1-62a8-4542-9481-f8cc26b16cad/volumes"
Sep 16 19:10:17.637947 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:17.637907    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:23.097639 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.097605    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"]
Sep 16 19:10:23.098009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.097948    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main"
Sep 16 19:10:23.098009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.097959    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main"
Sep 16 19:10:23.098009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.097968    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd882ea6-16d9-4e69-a81a-4b604b0214c1" containerName="main"
Sep 16 19:10:23.098009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.097973    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd882ea6-16d9-4e69-a81a-4b604b0214c1" containerName="main"
Sep 16 19:10:23.098009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.097989    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="storage-initializer"
Sep 16 19:10:23.098009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.097995    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="storage-initializer"
Sep 16 19:10:23.098009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.098002    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3934c1-62a8-4542-9481-f8cc26b16cad" containerName="main"
Sep 16 19:10:23.098009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.098007    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3934c1-62a8-4542-9481-f8cc26b16cad" containerName="main"
Sep 16 19:10:23.098268 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.098071    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3934c1-62a8-4542-9481-f8cc26b16cad" containerName="main"
Sep 16 19:10:23.098268 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.098079    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd882ea6-16d9-4e69-a81a-4b604b0214c1" containerName="main"
Sep 16 19:10:23.098268 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.098084    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6286a278-c312-4d1d-ab4d-1edfec9b667f" containerName="main"
Sep 16 19:10:23.101148 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.101130    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.104271 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.104246    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 19:10:23.104407 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.104246    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 16 19:10:23.104982 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.104960    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-q95xn\""
Sep 16 19:10:23.105070 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.104969    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:10:23.105070 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.105011    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 19:10:23.105070 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.105015    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 19:10:23.116512 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.116482    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"]
Sep 16 19:10:23.129864 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.129835    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:23.186109 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.186069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-dshm\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.186109 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.186106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8a8c99-fda0-4f8c-a166-e743166c6b69-tls-certs\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.186364 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.186131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-home\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.186364 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.186253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-tmp-dir\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.186364 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.186286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gllh9\" (UniqueName: \"kubernetes.io/projected/1a8a8c99-fda0-4f8c-a166-e743166c6b69-kube-api-access-gllh9\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.186364 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.186332    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-model-cache\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.186564 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.186367    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8a8c99-fda0-4f8c-a166-e743166c6b69-cabundle-cert\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.186564 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.186446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-kserve-provision-location\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.287622 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.287591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-kserve-provision-location\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.287784 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.287643    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-dshm\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.287784 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.287665    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8a8c99-fda0-4f8c-a166-e743166c6b69-tls-certs\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.287784 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.287690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-home\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.287784 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.287735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-tmp-dir\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.287784 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.287757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gllh9\" (UniqueName: \"kubernetes.io/projected/1a8a8c99-fda0-4f8c-a166-e743166c6b69-kube-api-access-gllh9\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.288051 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.287791    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-model-cache\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.288051 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.287829    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8a8c99-fda0-4f8c-a166-e743166c6b69-cabundle-cert\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.288164 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.288121    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-home\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.288219 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.288180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-tmp-dir\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.288430 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.288385    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-model-cache\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.288712 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.288690    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-kserve-provision-location\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.288818 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.288725    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8a8c99-fda0-4f8c-a166-e743166c6b69-cabundle-cert\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.290613 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.290567    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-dshm\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.290613 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.290605    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8a8c99-fda0-4f8c-a166-e743166c6b69-tls-certs\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.301791 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.301770    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gllh9\" (UniqueName: \"kubernetes.io/projected/1a8a8c99-fda0-4f8c-a166-e743166c6b69-kube-api-access-gllh9\") pod \"auth-disabled-test-kserve-7697cb85b4-b5pfp\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.308422 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.308396    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"]
Sep 16 19:10:23.312840 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.312820    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"
Sep 16 19:10:23.316047 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.316024    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-56l45\""
Sep 16 19:10:23.320698 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.320677    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"]
Sep 16 19:10:23.332678 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.332652    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:23.389043 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.388936    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0132945b-0cd4-4ae6-b72a-40c63c2caa86-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt\" (UID: \"0132945b-0cd4-4ae6-b72a-40c63c2caa86\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"
Sep 16 19:10:23.389206 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.389071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqjwt\" (UniqueName: \"kubernetes.io/projected/0132945b-0cd4-4ae6-b72a-40c63c2caa86-kube-api-access-qqjwt\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt\" (UID: \"0132945b-0cd4-4ae6-b72a-40c63c2caa86\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"
Sep 16 19:10:23.411919 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.411888    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:23.489941 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.489909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0132945b-0cd4-4ae6-b72a-40c63c2caa86-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt\" (UID: \"0132945b-0cd4-4ae6-b72a-40c63c2caa86\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"
Sep 16 19:10:23.490108 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.489981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqjwt\" (UniqueName: \"kubernetes.io/projected/0132945b-0cd4-4ae6-b72a-40c63c2caa86-kube-api-access-qqjwt\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt\" (UID: \"0132945b-0cd4-4ae6-b72a-40c63c2caa86\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"
Sep 16 19:10:23.492403 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.492378    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0132945b-0cd4-4ae6-b72a-40c63c2caa86-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt\" (UID: \"0132945b-0cd4-4ae6-b72a-40c63c2caa86\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"
Sep 16 19:10:23.498965 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.498942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqjwt\" (UniqueName: \"kubernetes.io/projected/0132945b-0cd4-4ae6-b72a-40c63c2caa86-kube-api-access-qqjwt\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt\" (UID: \"0132945b-0cd4-4ae6-b72a-40c63c2caa86\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"
Sep 16 19:10:23.517131 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.517107    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:23.619795 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.619755    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:23.623617 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.623597    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"
Sep 16 19:10:23.692631 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.692601    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"]
Sep 16 19:10:23.696352 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:10:23.696312    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8a8c99_fda0_4f8c_a166_e743166c6b69.slice/crio-bd7ba293fdafa5169ca945ae3d8561f615097e8dab0e6c0a70ee1b576819436c WatchSource:0}: Error finding container bd7ba293fdafa5169ca945ae3d8561f615097e8dab0e6c0a70ee1b576819436c: Status 404 returned error can't find the container with id bd7ba293fdafa5169ca945ae3d8561f615097e8dab0e6c0a70ee1b576819436c
Sep 16 19:10:23.902491 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:23.902445    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"]
Sep 16 19:10:23.907949 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:10:23.907911    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0132945b_0cd4_4ae6_b72a_40c63c2caa86.slice/crio-625087bbea86697152566d7478196d937b3182034bcde884cf61748c31c4a442 WatchSource:0}: Error finding container 625087bbea86697152566d7478196d937b3182034bcde884cf61748c31c4a442: Status 404 returned error can't find the container with id 625087bbea86697152566d7478196d937b3182034bcde884cf61748c31c4a442
Sep 16 19:10:24.082763 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:24.082726    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt" event={"ID":"0132945b-0cd4-4ae6-b72a-40c63c2caa86","Type":"ContainerStarted","Data":"a964467cae00dea80ab4c6c0358ff8e5ec470debfb49fcf76b949e348ac06eda"}
Sep 16 19:10:24.082955 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:24.082788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt" event={"ID":"0132945b-0cd4-4ae6-b72a-40c63c2caa86","Type":"ContainerStarted","Data":"625087bbea86697152566d7478196d937b3182034bcde884cf61748c31c4a442"}
Sep 16 19:10:24.082955 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:24.082816    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"
Sep 16 19:10:24.084241 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:24.084217    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" event={"ID":"1a8a8c99-fda0-4f8c-a166-e743166c6b69","Type":"ContainerStarted","Data":"b2aafa9d1ae6a13ba1466af9b78dd65a811fe463806dd5f8462b158ed94c8a68"}
Sep 16 19:10:24.084361 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:24.084257    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" event={"ID":"1a8a8c99-fda0-4f8c-a166-e743166c6b69","Type":"ContainerStarted","Data":"bd7ba293fdafa5169ca945ae3d8561f615097e8dab0e6c0a70ee1b576819436c"}
Sep 16 19:10:24.102520 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:24.102467    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt" podStartSLOduration=1.102447205 podStartE2EDuration="1.102447205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:24.099966346 +0000 UTC m=+1481.930574896" watchObservedRunningTime="2026-09-16 19:10:24.102447205 +0000 UTC m=+1481.933055753"
Sep 16 19:10:27.098414 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:27.098380    2582 generic.go:356] "Generic (PLEG): container finished" podID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerID="b2aafa9d1ae6a13ba1466af9b78dd65a811fe463806dd5f8462b158ed94c8a68" exitCode=0
Sep 16 19:10:27.098849 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:27.098457    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" event={"ID":"1a8a8c99-fda0-4f8c-a166-e743166c6b69","Type":"ContainerDied","Data":"b2aafa9d1ae6a13ba1466af9b78dd65a811fe463806dd5f8462b158ed94c8a68"}
Sep 16 19:10:28.106116 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:28.106069    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" event={"ID":"1a8a8c99-fda0-4f8c-a166-e743166c6b69","Type":"ContainerStarted","Data":"4930a4f0619d220d287b79db0e0b2345841aa40c7adb4f780848f6834b25f09a"}
Sep 16 19:10:28.130931 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:28.130799    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" podStartSLOduration=2.718690263 podStartE2EDuration="5.130758824s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.412068561s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:28.128686706 +0000 UTC m=+1485.959295252" watchObservedRunningTime="2026-09-16 19:10:28.130758824 +0000 UTC m=+1485.961367370"
Sep 16 19:10:28.412346 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:28.412223    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:33.412458 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:33.412409    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:10:33.414150 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:33.414118    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 19:10:35.828750 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:35.828714    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:35.925106 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:35.925067    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:41.942104 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:41.942068    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:43.412604 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:43.412546    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 19:10:47.373231 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:47.373191    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:47.583675 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:47.583642    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:47.979057 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:47.979019    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:48.079483 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:48.079452    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:53.412845 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:53.412799    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 19:10:55.090946 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:55.090907    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"
Sep 16 19:10:59.680883 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:10:59.680845    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:03.412764 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:11:03.412655    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 19:11:13.413205 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:11:13.413144    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 19:11:23.413058 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:11:23.413001    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 19:11:26.898814 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:11:26.898778    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:26.982217 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:11:26.982165    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:33.412594 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:11:33.412527    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 19:11:43.412539 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:11:43.412489    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 19:11:53.413295 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:11:53.413235    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 19:11:58.643175 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:11:58.643129    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:01.201332 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:01.201283    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:03.413073 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:03.413014    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 19:12:07.319728 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:07.319693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:07.421543 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:07.421489    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:09.576946 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:09.576899    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:13.413123 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:13.413061    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 19:12:16.951853 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:16.951800    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:17.052540 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:17.052506    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:23.122308 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:23.122271    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:23.413371 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:23.413249    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 19:12:24.596182 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:24.596137    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:24.695623 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:24.695564    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:29.059458 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:29.059429    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:29.144847 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:29.144810    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:33.422776 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:33.422689    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:12:33.430914 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:33.430885    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:12:39.846561 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:39.846528    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:46.355376 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:46.355337    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"]
Sep 16 19:12:46.358238 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:46.358208    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:46.358447 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:46.358237    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"]
Sep 16 19:12:46.452077 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:46.452044    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:49.556776 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:49.556742    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:55.090313 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:55.090280    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt" podUID="0132945b-0cd4-4ae6-b72a-40c63c2caa86" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:12:59.349031 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:59.348984    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:59.426835 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:12:59.426797    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:01.378069 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:01.378004    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt" podUID="0132945b-0cd4-4ae6-b72a-40c63c2caa86" containerName="main" containerID="cri-o://a964467cae00dea80ab4c6c0358ff8e5ec470debfb49fcf76b949e348ac06eda" gracePeriod=45
Sep 16 19:13:01.383690 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:01.383654    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main" containerID="cri-o://4930a4f0619d220d287b79db0e0b2345841aa40c7adb4f780848f6834b25f09a" gracePeriod=45
Sep 16 19:13:02.140792 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.140764    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:13:02.233646 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.233538    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-tmp-dir\") pod \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") "
Sep 16 19:13:02.233646 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.233596    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gllh9\" (UniqueName: \"kubernetes.io/projected/1a8a8c99-fda0-4f8c-a166-e743166c6b69-kube-api-access-gllh9\") pod \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") "
Sep 16 19:13:02.233646 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.233624    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-kserve-provision-location\") pod \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") "
Sep 16 19:13:02.233938 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.233653    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-model-cache\") pod \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") "
Sep 16 19:13:02.233938 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.233683    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8a8c99-fda0-4f8c-a166-e743166c6b69-tls-certs\") pod \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") "
Sep 16 19:13:02.233938 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.233764    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8a8c99-fda0-4f8c-a166-e743166c6b69-cabundle-cert\") pod \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") "
Sep 16 19:13:02.233938 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.233806    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-home\") pod \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") "
Sep 16 19:13:02.233938 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.233859    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-dshm\") pod \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\" (UID: \"1a8a8c99-fda0-4f8c-a166-e743166c6b69\") "
Sep 16 19:13:02.234195 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.234168    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8a8c99-fda0-4f8c-a166-e743166c6b69-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a8a8c99-fda0-4f8c-a166-e743166c6b69" (UID: "1a8a8c99-fda0-4f8c-a166-e743166c6b69"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:02.234331 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.234304    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8a8c99-fda0-4f8c-a166-e743166c6b69-cabundle-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:13:02.234533 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.234506    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-home" (OuterVolumeSpecName: "home") pod "1a8a8c99-fda0-4f8c-a166-e743166c6b69" (UID: "1a8a8c99-fda0-4f8c-a166-e743166c6b69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:02.234637 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.234599    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a8a8c99-fda0-4f8c-a166-e743166c6b69" (UID: "1a8a8c99-fda0-4f8c-a166-e743166c6b69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:02.236519 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.236486    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8a8c99-fda0-4f8c-a166-e743166c6b69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a8a8c99-fda0-4f8c-a166-e743166c6b69" (UID: "1a8a8c99-fda0-4f8c-a166-e743166c6b69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:02.236666 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.236650    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-dshm" (OuterVolumeSpecName: "dshm") pod "1a8a8c99-fda0-4f8c-a166-e743166c6b69" (UID: "1a8a8c99-fda0-4f8c-a166-e743166c6b69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:02.236756 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.236667    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8a8c99-fda0-4f8c-a166-e743166c6b69-kube-api-access-gllh9" (OuterVolumeSpecName: "kube-api-access-gllh9") pod "1a8a8c99-fda0-4f8c-a166-e743166c6b69" (UID: "1a8a8c99-fda0-4f8c-a166-e743166c6b69"). InnerVolumeSpecName "kube-api-access-gllh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:02.246982 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.246953    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a8a8c99-fda0-4f8c-a166-e743166c6b69" (UID: "1a8a8c99-fda0-4f8c-a166-e743166c6b69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:02.264805 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.264777    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a8a8c99-fda0-4f8c-a166-e743166c6b69" (UID: "1a8a8c99-fda0-4f8c-a166-e743166c6b69"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:02.335294 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.335262    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8a8c99-fda0-4f8c-a166-e743166c6b69-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:13:02.335294 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.335289    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:13:02.335294 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.335297    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:13:02.335491 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.335304    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:13:02.335491 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.335312    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gllh9\" (UniqueName: \"kubernetes.io/projected/1a8a8c99-fda0-4f8c-a166-e743166c6b69-kube-api-access-gllh9\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:13:02.335491 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.335329    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:13:02.335491 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.335338    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8a8c99-fda0-4f8c-a166-e743166c6b69-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:13:02.929773 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.929735    2582 generic.go:356] "Generic (PLEG): container finished" podID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerID="4930a4f0619d220d287b79db0e0b2345841aa40c7adb4f780848f6834b25f09a" exitCode=0
Sep 16 19:13:02.930191 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.929787    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"
Sep 16 19:13:02.930191 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.929861    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" event={"ID":"1a8a8c99-fda0-4f8c-a166-e743166c6b69","Type":"ContainerDied","Data":"4930a4f0619d220d287b79db0e0b2345841aa40c7adb4f780848f6834b25f09a"}
Sep 16 19:13:02.930191 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.929915    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp" event={"ID":"1a8a8c99-fda0-4f8c-a166-e743166c6b69","Type":"ContainerDied","Data":"bd7ba293fdafa5169ca945ae3d8561f615097e8dab0e6c0a70ee1b576819436c"}
Sep 16 19:13:02.930637 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.929939    2582 scope.go:117] "RemoveContainer" containerID="4930a4f0619d220d287b79db0e0b2345841aa40c7adb4f780848f6834b25f09a"
Sep 16 19:13:02.939355 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.939334    2582 scope.go:117] "RemoveContainer" containerID="b2aafa9d1ae6a13ba1466af9b78dd65a811fe463806dd5f8462b158ed94c8a68"
Sep 16 19:13:02.947170 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.947146    2582 scope.go:117] "RemoveContainer" containerID="4930a4f0619d220d287b79db0e0b2345841aa40c7adb4f780848f6834b25f09a"
Sep 16 19:13:02.947486 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:13:02.947467    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4930a4f0619d220d287b79db0e0b2345841aa40c7adb4f780848f6834b25f09a\": container with ID starting with 4930a4f0619d220d287b79db0e0b2345841aa40c7adb4f780848f6834b25f09a not found: ID does not exist" containerID="4930a4f0619d220d287b79db0e0b2345841aa40c7adb4f780848f6834b25f09a"
Sep 16 19:13:02.947593 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.947499    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4930a4f0619d220d287b79db0e0b2345841aa40c7adb4f780848f6834b25f09a"} err="failed to get container status \"4930a4f0619d220d287b79db0e0b2345841aa40c7adb4f780848f6834b25f09a\": rpc error: code = NotFound desc = could not find container \"4930a4f0619d220d287b79db0e0b2345841aa40c7adb4f780848f6834b25f09a\": container with ID starting with 4930a4f0619d220d287b79db0e0b2345841aa40c7adb4f780848f6834b25f09a not found: ID does not exist"
Sep 16 19:13:02.947593 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.947522    2582 scope.go:117] "RemoveContainer" containerID="b2aafa9d1ae6a13ba1466af9b78dd65a811fe463806dd5f8462b158ed94c8a68"
Sep 16 19:13:02.947834 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:13:02.947817    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2aafa9d1ae6a13ba1466af9b78dd65a811fe463806dd5f8462b158ed94c8a68\": container with ID starting with b2aafa9d1ae6a13ba1466af9b78dd65a811fe463806dd5f8462b158ed94c8a68 not found: ID does not exist" containerID="b2aafa9d1ae6a13ba1466af9b78dd65a811fe463806dd5f8462b158ed94c8a68"
Sep 16 19:13:02.947876 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.947841    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2aafa9d1ae6a13ba1466af9b78dd65a811fe463806dd5f8462b158ed94c8a68"} err="failed to get container status \"b2aafa9d1ae6a13ba1466af9b78dd65a811fe463806dd5f8462b158ed94c8a68\": rpc error: code = NotFound desc = could not find container \"b2aafa9d1ae6a13ba1466af9b78dd65a811fe463806dd5f8462b158ed94c8a68\": container with ID starting with b2aafa9d1ae6a13ba1466af9b78dd65a811fe463806dd5f8462b158ed94c8a68 not found: ID does not exist"
Sep 16 19:13:02.951024 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.951004    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"]
Sep 16 19:13:02.955502 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:02.955481    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7697cb85b4-b5pfp"]
Sep 16 19:13:04.703915 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:04.703882    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" path="/var/lib/kubelet/pods/1a8a8c99-fda0-4f8c-a166-e743166c6b69/volumes"
Sep 16 19:13:05.090697 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:05.090654    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt" podUID="0132945b-0cd4-4ae6-b72a-40c63c2caa86" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:13:15.091076 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:15.091036    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt" podUID="0132945b-0cd4-4ae6-b72a-40c63c2caa86" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:13:15.091663 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:15.091156    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"
Sep 16 19:13:25.090997 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:25.090952    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt" podUID="0132945b-0cd4-4ae6-b72a-40c63c2caa86" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:13:31.622947 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:31.622920    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"
Sep 16 19:13:31.716483 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:31.716395    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqjwt\" (UniqueName: \"kubernetes.io/projected/0132945b-0cd4-4ae6-b72a-40c63c2caa86-kube-api-access-qqjwt\") pod \"0132945b-0cd4-4ae6-b72a-40c63c2caa86\" (UID: \"0132945b-0cd4-4ae6-b72a-40c63c2caa86\") "
Sep 16 19:13:31.716483 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:31.716455    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0132945b-0cd4-4ae6-b72a-40c63c2caa86-tls-certs\") pod \"0132945b-0cd4-4ae6-b72a-40c63c2caa86\" (UID: \"0132945b-0cd4-4ae6-b72a-40c63c2caa86\") "
Sep 16 19:13:31.718625 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:31.718590    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0132945b-0cd4-4ae6-b72a-40c63c2caa86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0132945b-0cd4-4ae6-b72a-40c63c2caa86" (UID: "0132945b-0cd4-4ae6-b72a-40c63c2caa86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:31.718625 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:31.718619    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0132945b-0cd4-4ae6-b72a-40c63c2caa86-kube-api-access-qqjwt" (OuterVolumeSpecName: "kube-api-access-qqjwt") pod "0132945b-0cd4-4ae6-b72a-40c63c2caa86" (UID: "0132945b-0cd4-4ae6-b72a-40c63c2caa86"). InnerVolumeSpecName "kube-api-access-qqjwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:31.817442 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:31.817412    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0132945b-0cd4-4ae6-b72a-40c63c2caa86-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:13:31.817562 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:31.817500    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqjwt\" (UniqueName: \"kubernetes.io/projected/0132945b-0cd4-4ae6-b72a-40c63c2caa86-kube-api-access-qqjwt\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:13:32.068374 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:32.068346    2582 generic.go:356] "Generic (PLEG): container finished" podID="0132945b-0cd4-4ae6-b72a-40c63c2caa86" containerID="a964467cae00dea80ab4c6c0358ff8e5ec470debfb49fcf76b949e348ac06eda" exitCode=0
Sep 16 19:13:32.068506 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:32.068387    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"
Sep 16 19:13:32.068506 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:32.068388    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt" event={"ID":"0132945b-0cd4-4ae6-b72a-40c63c2caa86","Type":"ContainerDied","Data":"a964467cae00dea80ab4c6c0358ff8e5ec470debfb49fcf76b949e348ac06eda"}
Sep 16 19:13:32.068604 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:32.068535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt" event={"ID":"0132945b-0cd4-4ae6-b72a-40c63c2caa86","Type":"ContainerDied","Data":"625087bbea86697152566d7478196d937b3182034bcde884cf61748c31c4a442"}
Sep 16 19:13:32.068604 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:32.068562    2582 scope.go:117] "RemoveContainer" containerID="a964467cae00dea80ab4c6c0358ff8e5ec470debfb49fcf76b949e348ac06eda"
Sep 16 19:13:32.077707 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:32.077691    2582 scope.go:117] "RemoveContainer" containerID="a964467cae00dea80ab4c6c0358ff8e5ec470debfb49fcf76b949e348ac06eda"
Sep 16 19:13:32.077963 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:13:32.077945    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a964467cae00dea80ab4c6c0358ff8e5ec470debfb49fcf76b949e348ac06eda\": container with ID starting with a964467cae00dea80ab4c6c0358ff8e5ec470debfb49fcf76b949e348ac06eda not found: ID does not exist" containerID="a964467cae00dea80ab4c6c0358ff8e5ec470debfb49fcf76b949e348ac06eda"
Sep 16 19:13:32.078034 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:32.077969    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a964467cae00dea80ab4c6c0358ff8e5ec470debfb49fcf76b949e348ac06eda"} err="failed to get container status \"a964467cae00dea80ab4c6c0358ff8e5ec470debfb49fcf76b949e348ac06eda\": rpc error: code = NotFound desc = could not find container \"a964467cae00dea80ab4c6c0358ff8e5ec470debfb49fcf76b949e348ac06eda\": container with ID starting with a964467cae00dea80ab4c6c0358ff8e5ec470debfb49fcf76b949e348ac06eda not found: ID does not exist"
Sep 16 19:13:32.092321 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:32.092300    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"]
Sep 16 19:13:32.096553 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:32.096530    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hbfpt"]
Sep 16 19:13:32.703276 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:32.703251    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0132945b-0cd4-4ae6-b72a-40c63c2caa86" path="/var/lib/kubelet/pods/0132945b-0cd4-4ae6-b72a-40c63c2caa86/volumes"
Sep 16 19:13:36.436320 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:36.436283    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:36.536862 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:36.536817    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:36.636665 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:36.636631    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:42.361987 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:42.361954    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:46.354916 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:46.354870    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:48.213551 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:48.213510    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:48.315404 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:48.315367    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:49.423117 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:49.423077    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:49.519935 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:49.519897    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:55.457247 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:13:55.457214    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:01.158886 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:01.158853    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:01.259973 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:01.259940    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:33.684773 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.684742    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"]
Sep 16 19:14:33.685211 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.685193    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="storage-initializer"
Sep 16 19:14:33.685260 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.685216    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="storage-initializer"
Sep 16 19:14:33.685260 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.685242    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main"
Sep 16 19:14:33.685260 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.685253    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main"
Sep 16 19:14:33.685374 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.685274    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0132945b-0cd4-4ae6-b72a-40c63c2caa86" containerName="main"
Sep 16 19:14:33.685374 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.685283    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0132945b-0cd4-4ae6-b72a-40c63c2caa86" containerName="main"
Sep 16 19:14:33.685374 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.685372    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8a8c99-fda0-4f8c-a166-e743166c6b69" containerName="main"
Sep 16 19:14:33.685476 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.685390    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0132945b-0cd4-4ae6-b72a-40c63c2caa86" containerName="main"
Sep 16 19:14:33.688586 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.688550    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.691679 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.691658    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 16 19:14:33.692067 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.692048    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 16 19:14:33.693311 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.693294    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-zprnr\""
Sep 16 19:14:33.693416 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.693366    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 16 19:14:33.696826 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.696807    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"]
Sep 16 19:14:33.703336 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.703314    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.703444 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.703350    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-home\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.703444 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.703386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ed0bf3-e77e-493f-bb59-32fba37d1b53-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.703565 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.703479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn47h\" (UniqueName: \"kubernetes.io/projected/98ed0bf3-e77e-493f-bb59-32fba37d1b53-kube-api-access-gn47h\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.703565 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.703538    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-dshm\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.703674 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.703596    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.716679 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.716645    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:33.804891 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.804869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn47h\" (UniqueName: \"kubernetes.io/projected/98ed0bf3-e77e-493f-bb59-32fba37d1b53-kube-api-access-gn47h\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.804974 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.804908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-dshm\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.804974 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.804924    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.805055 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.805037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.805091 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.805073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-home\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.805132 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.805114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ed0bf3-e77e-493f-bb59-32fba37d1b53-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.805251 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.805238    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.805355 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.805339    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.805433 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.805413    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-home\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.807193 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.807177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-dshm\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.807501 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.807476    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ed0bf3-e77e-493f-bb59-32fba37d1b53-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.814266 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.814243    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn47h\" (UniqueName: \"kubernetes.io/projected/98ed0bf3-e77e-493f-bb59-32fba37d1b53-kube-api-access-gn47h\") pod \"canary-v2-kserve-7f65dbbccc-xrngc\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:33.999270 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:33.999161    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:34.277717 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:34.277682    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"]
Sep 16 19:14:34.279996 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:14:34.279964    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ed0bf3_e77e_493f_bb59_32fba37d1b53.slice/crio-3a62ee6f28e6271be1ee6908cd0a6b125f7e3c419a725479fa9edd8c8d89a6ee WatchSource:0}: Error finding container 3a62ee6f28e6271be1ee6908cd0a6b125f7e3c419a725479fa9edd8c8d89a6ee: Status 404 returned error can't find the container with id 3a62ee6f28e6271be1ee6908cd0a6b125f7e3c419a725479fa9edd8c8d89a6ee
Sep 16 19:14:34.282003 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:34.281982    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:14:34.352791 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:34.352739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc" event={"ID":"98ed0bf3-e77e-493f-bb59-32fba37d1b53","Type":"ContainerStarted","Data":"3a62ee6f28e6271be1ee6908cd0a6b125f7e3c419a725479fa9edd8c8d89a6ee"}
Sep 16 19:14:36.364666 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:36.364627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc" event={"ID":"98ed0bf3-e77e-493f-bb59-32fba37d1b53","Type":"ContainerStarted","Data":"9303b989cd6ebdd5a637c6e3736182503581630f1da5604f066cb842e4d3ff09"}
Sep 16 19:14:36.389601 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:36.388888    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc" podStartSLOduration=2.374114165 podStartE2EDuration="3.388865808s" totalImagesPullingTime="1.014751643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:36.386087372 +0000 UTC m=+1734.216695928" watchObservedRunningTime="2026-09-16 19:14:36.388865808 +0000 UTC m=+1734.219474355"
Sep 16 19:14:36.999600 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:36.999526    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:43.999873 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:43.999827    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:44.012715 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:44.012686    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:44.412918 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:44.412877    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:14:54.323463 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.323410    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"]
Sep 16 19:14:54.327640 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.327608    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.331311 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.331265    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:14:54.331505 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.331266    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 19:14:54.331505 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.331268    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 19:14:54.331661 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.331645    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 19:14:54.332833 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.332813    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-k2pfb\""
Sep 16 19:14:54.332940 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.332885    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 19:14:54.338598 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.338540    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"]
Sep 16 19:14:54.373189 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.373143    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:54.409759 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.409710    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-model-cache\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.409759 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.409759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvsxv\" (UniqueName: \"kubernetes.io/projected/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-kube-api-access-wvsxv\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.410066 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.409826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.410066 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.409866    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-dshm\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.410066 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.409917    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.410066 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.409952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.410066 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.409980    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.410066 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.410049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-home\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.510881 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.510835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.511124 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.510903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-home\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.511124 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.510956    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-model-cache\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.511124 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.510979    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvsxv\" (UniqueName: \"kubernetes.io/projected/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-kube-api-access-wvsxv\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.511124 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.511005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.511124 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.511037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-dshm\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.511124 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.511082    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.511124 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.511115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.511645 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.511618    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-model-cache\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.511745 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.511662    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.511789 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.511739    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-home\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.511789 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.511774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.512010 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.511993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.513752 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.513731    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-dshm\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.513930 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.513913    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.521369 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.521347    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvsxv\" (UniqueName: \"kubernetes.io/projected/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-kube-api-access-wvsxv\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.619486 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.619389    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"]
Sep 16 19:14:54.623120 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.623100    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"
Sep 16 19:14:54.626532 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.626510    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-brm6h\""
Sep 16 19:14:54.633668 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.633633    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"]
Sep 16 19:14:54.641765 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.641732    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:14:54.643002 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.642972    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:54.713128 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.713089    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rfsk\" (UniqueName: \"kubernetes.io/projected/b96fa0ac-f519-4458-84a6-ae8e24c04843-kube-api-access-2rfsk\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd\" (UID: \"b96fa0ac-f519-4458-84a6-ae8e24c04843\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"
Sep 16 19:14:54.713288 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.713242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96fa0ac-f519-4458-84a6-ae8e24c04843-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd\" (UID: \"b96fa0ac-f519-4458-84a6-ae8e24c04843\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"
Sep 16 19:14:54.741800 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.741741    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:54.814239 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.814177    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96fa0ac-f519-4458-84a6-ae8e24c04843-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd\" (UID: \"b96fa0ac-f519-4458-84a6-ae8e24c04843\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"
Sep 16 19:14:54.814515 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.814272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfsk\" (UniqueName: \"kubernetes.io/projected/b96fa0ac-f519-4458-84a6-ae8e24c04843-kube-api-access-2rfsk\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd\" (UID: \"b96fa0ac-f519-4458-84a6-ae8e24c04843\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"
Sep 16 19:14:54.817333 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.817308    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96fa0ac-f519-4458-84a6-ae8e24c04843-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd\" (UID: \"b96fa0ac-f519-4458-84a6-ae8e24c04843\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"
Sep 16 19:14:54.829007 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.828970    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rfsk\" (UniqueName: \"kubernetes.io/projected/b96fa0ac-f519-4458-84a6-ae8e24c04843-kube-api-access-2rfsk\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd\" (UID: \"b96fa0ac-f519-4458-84a6-ae8e24c04843\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"
Sep 16 19:14:54.935967 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.935839    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"
Sep 16 19:14:54.942190 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:54.942160    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"]
Sep 16 19:14:54.946995 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:14:54.946965    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c0b8ad_8303_4f96_8b99_54750cbc13c1.slice/crio-466b26773209899c54492391780a2855a41242dfef9f6fe2fc5eb601d49f84c0 WatchSource:0}: Error finding container 466b26773209899c54492391780a2855a41242dfef9f6fe2fc5eb601d49f84c0: Status 404 returned error can't find the container with id 466b26773209899c54492391780a2855a41242dfef9f6fe2fc5eb601d49f84c0
Sep 16 19:14:55.227477 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:55.227388    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"]
Sep 16 19:14:55.230409 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:14:55.230374    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96fa0ac_f519_4458_84a6_ae8e24c04843.slice/crio-11bdb40b5ee255f76e0858557be3b02af8c32f1726d4bcb1d1df76a2b56a58bd WatchSource:0}: Error finding container 11bdb40b5ee255f76e0858557be3b02af8c32f1726d4bcb1d1df76a2b56a58bd: Status 404 returned error can't find the container with id 11bdb40b5ee255f76e0858557be3b02af8c32f1726d4bcb1d1df76a2b56a58bd
Sep 16 19:14:55.458544 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:55.458483    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd" event={"ID":"b96fa0ac-f519-4458-84a6-ae8e24c04843","Type":"ContainerStarted","Data":"aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3"}
Sep 16 19:14:55.459122 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:55.458606    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"
Sep 16 19:14:55.459122 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:55.458640    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd" event={"ID":"b96fa0ac-f519-4458-84a6-ae8e24c04843","Type":"ContainerStarted","Data":"11bdb40b5ee255f76e0858557be3b02af8c32f1726d4bcb1d1df76a2b56a58bd"}
Sep 16 19:14:55.460497 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:55.460464    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf" event={"ID":"b8c0b8ad-8303-4f96-8b99-54750cbc13c1","Type":"ContainerStarted","Data":"0e7e81d4a12fae262f39d12f6096acf2be8d7c49b5f61d2ff46053ad1fa53427"}
Sep 16 19:14:55.460654 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:55.460522    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf" event={"ID":"b8c0b8ad-8303-4f96-8b99-54750cbc13c1","Type":"ContainerStarted","Data":"466b26773209899c54492391780a2855a41242dfef9f6fe2fc5eb601d49f84c0"}
Sep 16 19:14:55.485512 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:55.485383    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd" podStartSLOduration=1.485364151 podStartE2EDuration="1.485364151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:55.482467294 +0000 UTC m=+1753.313075838" watchObservedRunningTime="2026-09-16 19:14:55.485364151 +0000 UTC m=+1753.315972696"
Sep 16 19:14:58.477315 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:58.477272    2582 generic.go:356] "Generic (PLEG): container finished" podID="b8c0b8ad-8303-4f96-8b99-54750cbc13c1" containerID="0e7e81d4a12fae262f39d12f6096acf2be8d7c49b5f61d2ff46053ad1fa53427" exitCode=0
Sep 16 19:14:58.477823 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:58.477350    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf" event={"ID":"b8c0b8ad-8303-4f96-8b99-54750cbc13c1","Type":"ContainerDied","Data":"0e7e81d4a12fae262f39d12f6096acf2be8d7c49b5f61d2ff46053ad1fa53427"}
Sep 16 19:14:59.485227 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:59.485179    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf" event={"ID":"b8c0b8ad-8303-4f96-8b99-54750cbc13c1","Type":"ContainerStarted","Data":"dfc23bb3041a477864b7b4dbfc0b256e395654878a6984f7f638add21904be2b"}
Sep 16 19:14:59.513750 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:59.513676    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf" podStartSLOduration=2.9489919799999997 podStartE2EDuration="5.513657335s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.564665355s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:59.510834536 +0000 UTC m=+1757.341443082" watchObservedRunningTime="2026-09-16 19:14:59.513657335 +0000 UTC m=+1757.344265983"
Sep 16 19:14:59.642764 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:14:59.642723    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:15:04.642442 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:15:04.642384    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:15:04.655739 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:15:04.655707    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:15:05.525676 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:15:05.525641    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:15:26.467394 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:15:26.467358    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"
Sep 16 19:15:31.373492 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:15:31.373447    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:38.678662 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:15:38.678521    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:38.752525 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:15:38.752486    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:21.548183 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:21.548144    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"]
Sep 16 19:16:21.551500 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:21.551455    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:21.555370 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:21.555338    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"]
Sep 16 19:16:21.561030 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:16:21.560991    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:16:21.561030 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:16:21.561030 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:16:21.561030 ip-10-0-142-173 kubenswrapper[2582]:  > containerID="dfc23bb3041a477864b7b4dbfc0b256e395654878a6984f7f638add21904be2b" cmd=["/bin/sleep","15"]
Sep 16 19:16:21.561324 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:21.561052    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf" message="" err=<
Sep 16 19:16:21.561324 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:16:21.561324 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:16:21.561324 ip-10-0-142-173 kubenswrapper[2582]:  >
Sep 16 19:16:21.561324 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:16:21.561070    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:16:21.561324 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:16:21.561324 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:16:21.561324 ip-10-0-142-173 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf" podUID="b8c0b8ad-8303-4f96-8b99-54750cbc13c1" containerName="main" containerID="cri-o://dfc23bb3041a477864b7b4dbfc0b256e395654878a6984f7f638add21904be2b"
Sep 16 19:16:21.561324 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:21.561112    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf" podUID="b8c0b8ad-8303-4f96-8b99-54750cbc13c1" containerName="main" containerID="cri-o://dfc23bb3041a477864b7b4dbfc0b256e395654878a6984f7f638add21904be2b" gracePeriod=60
Sep 16 19:16:21.625269 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:21.625232    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:21.898743 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:21.898713    2582 generic.go:356] "Generic (PLEG): container finished" podID="b8c0b8ad-8303-4f96-8b99-54750cbc13c1" containerID="dfc23bb3041a477864b7b4dbfc0b256e395654878a6984f7f638add21904be2b" exitCode=0
Sep 16 19:16:21.898885 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:21.898784    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf" event={"ID":"b8c0b8ad-8303-4f96-8b99-54750cbc13c1","Type":"ContainerDied","Data":"dfc23bb3041a477864b7b4dbfc0b256e395654878a6984f7f638add21904be2b"}
Sep 16 19:16:21.926257 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:21.926227    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:16:22.061810 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.061774    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-tls-certs\") pod \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") "
Sep 16 19:16:22.061810 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.061809    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-home\") pod \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") "
Sep 16 19:16:22.062063 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.061845    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-model-cache\") pod \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") "
Sep 16 19:16:22.062063 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.061863    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-cabundle-cert\") pod \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") "
Sep 16 19:16:22.062063 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.061891    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-tmp-dir\") pod \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") "
Sep 16 19:16:22.062063 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.061936    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvsxv\" (UniqueName: \"kubernetes.io/projected/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-kube-api-access-wvsxv\") pod \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") "
Sep 16 19:16:22.062063 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.061975    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-dshm\") pod \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") "
Sep 16 19:16:22.062063 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.062034    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-kserve-provision-location\") pod \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\" (UID: \"b8c0b8ad-8303-4f96-8b99-54750cbc13c1\") "
Sep 16 19:16:22.062396 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.062068    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "b8c0b8ad-8303-4f96-8b99-54750cbc13c1" (UID: "b8c0b8ad-8303-4f96-8b99-54750cbc13c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:22.062396 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.062081    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-home" (OuterVolumeSpecName: "home") pod "b8c0b8ad-8303-4f96-8b99-54750cbc13c1" (UID: "b8c0b8ad-8303-4f96-8b99-54750cbc13c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:22.062396 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.062150    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b8c0b8ad-8303-4f96-8b99-54750cbc13c1" (UID: "b8c0b8ad-8303-4f96-8b99-54750cbc13c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:22.062396 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.062232    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b8c0b8ad-8303-4f96-8b99-54750cbc13c1" (UID: "b8c0b8ad-8303-4f96-8b99-54750cbc13c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:22.062702 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.062406    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.062702 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.062424    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.062702 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.062439    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-cabundle-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.062702 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.062470    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.064507 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.064480    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-dshm" (OuterVolumeSpecName: "dshm") pod "b8c0b8ad-8303-4f96-8b99-54750cbc13c1" (UID: "b8c0b8ad-8303-4f96-8b99-54750cbc13c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:22.064778 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.064757    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-kube-api-access-wvsxv" (OuterVolumeSpecName: "kube-api-access-wvsxv") pod "b8c0b8ad-8303-4f96-8b99-54750cbc13c1" (UID: "b8c0b8ad-8303-4f96-8b99-54750cbc13c1"). InnerVolumeSpecName "kube-api-access-wvsxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:22.065073 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.065053    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8c0b8ad-8303-4f96-8b99-54750cbc13c1" (UID: "b8c0b8ad-8303-4f96-8b99-54750cbc13c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:22.086655 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.086533    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8c0b8ad-8303-4f96-8b99-54750cbc13c1" (UID: "b8c0b8ad-8303-4f96-8b99-54750cbc13c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:22.163249 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.163213    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.163249 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.163248    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvsxv\" (UniqueName: \"kubernetes.io/projected/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-kube-api-access-wvsxv\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.163518 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.163259    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.163518 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.163271    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8c0b8ad-8303-4f96-8b99-54750cbc13c1-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.905908 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.905881    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"
Sep 16 19:16:22.905908 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.905902    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf" event={"ID":"b8c0b8ad-8303-4f96-8b99-54750cbc13c1","Type":"ContainerDied","Data":"466b26773209899c54492391780a2855a41242dfef9f6fe2fc5eb601d49f84c0"}
Sep 16 19:16:22.906414 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.905961    2582 scope.go:117] "RemoveContainer" containerID="dfc23bb3041a477864b7b4dbfc0b256e395654878a6984f7f638add21904be2b"
Sep 16 19:16:22.915667 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.915553    2582 scope.go:117] "RemoveContainer" containerID="0e7e81d4a12fae262f39d12f6096acf2be8d7c49b5f61d2ff46053ad1fa53427"
Sep 16 19:16:22.928308 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.928269    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"]
Sep 16 19:16:22.931170 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:22.931141    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-hkhtf"]
Sep 16 19:16:24.704307 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:24.704268    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8c0b8ad-8303-4f96-8b99-54750cbc13c1" path="/var/lib/kubelet/pods/b8c0b8ad-8303-4f96-8b99-54750cbc13c1/volumes"
Sep 16 19:16:26.468170 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:26.468120    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd" podUID="b96fa0ac-f519-4458-84a6-ae8e24c04843" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:16:36.467993 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:36.467952    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd" podUID="b96fa0ac-f519-4458-84a6-ae8e24c04843" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:16:36.578010 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:36.577956    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd" podUID="b96fa0ac-f519-4458-84a6-ae8e24c04843" containerName="main" containerID="cri-o://aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3" gracePeriod=45
Sep 16 19:16:46.467473 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:46.467429    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd" podUID="b96fa0ac-f519-4458-84a6-ae8e24c04843" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:16:46.467908 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:46.467567    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"
Sep 16 19:16:56.467952 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:16:56.467907    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd" podUID="b96fa0ac-f519-4458-84a6-ae8e24c04843" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:17:01.653722 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:01.653679    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:02.802931 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:02.802888    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:05.448257 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:05.448196    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:05.566406 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:05.566363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:05.575469 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:05.575440    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"]
Sep 16 19:17:05.655740 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:05.655694    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:05.856947 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:05.856911    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:06.467664 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:06.467625    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd" podUID="b96fa0ac-f519-4458-84a6-ae8e24c04843" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:17:06.613531 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:17:06.613489    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96fa0ac_f519_4458_84a6_ae8e24c04843.slice/crio-aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96fa0ac_f519_4458_84a6_ae8e24c04843.slice/crio-conmon-aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:17:06.835614 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:06.835585    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"
Sep 16 19:17:06.968685 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:06.968634    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96fa0ac-f519-4458-84a6-ae8e24c04843-tls-certs\") pod \"b96fa0ac-f519-4458-84a6-ae8e24c04843\" (UID: \"b96fa0ac-f519-4458-84a6-ae8e24c04843\") "
Sep 16 19:17:06.968874 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:06.968731    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rfsk\" (UniqueName: \"kubernetes.io/projected/b96fa0ac-f519-4458-84a6-ae8e24c04843-kube-api-access-2rfsk\") pod \"b96fa0ac-f519-4458-84a6-ae8e24c04843\" (UID: \"b96fa0ac-f519-4458-84a6-ae8e24c04843\") "
Sep 16 19:17:06.971054 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:06.971018    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96fa0ac-f519-4458-84a6-ae8e24c04843-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b96fa0ac-f519-4458-84a6-ae8e24c04843" (UID: "b96fa0ac-f519-4458-84a6-ae8e24c04843"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:06.971054 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:06.971039    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96fa0ac-f519-4458-84a6-ae8e24c04843-kube-api-access-2rfsk" (OuterVolumeSpecName: "kube-api-access-2rfsk") pod "b96fa0ac-f519-4458-84a6-ae8e24c04843" (UID: "b96fa0ac-f519-4458-84a6-ae8e24c04843"). InnerVolumeSpecName "kube-api-access-2rfsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:07.070363 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:07.070323    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rfsk\" (UniqueName: \"kubernetes.io/projected/b96fa0ac-f519-4458-84a6-ae8e24c04843-kube-api-access-2rfsk\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:17:07.070363 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:07.070360    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96fa0ac-f519-4458-84a6-ae8e24c04843-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:17:07.112917 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:07.112885    2582 generic.go:356] "Generic (PLEG): container finished" podID="b96fa0ac-f519-4458-84a6-ae8e24c04843" containerID="aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3" exitCode=0
Sep 16 19:17:07.113068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:07.112928    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"
Sep 16 19:17:07.113068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:07.112966    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd" event={"ID":"b96fa0ac-f519-4458-84a6-ae8e24c04843","Type":"ContainerDied","Data":"aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3"}
Sep 16 19:17:07.113068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:07.113016    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd" event={"ID":"b96fa0ac-f519-4458-84a6-ae8e24c04843","Type":"ContainerDied","Data":"11bdb40b5ee255f76e0858557be3b02af8c32f1726d4bcb1d1df76a2b56a58bd"}
Sep 16 19:17:07.113068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:07.113037    2582 scope.go:117] "RemoveContainer" containerID="aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3"
Sep 16 19:17:07.122680 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:07.122664    2582 scope.go:117] "RemoveContainer" containerID="aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3"
Sep 16 19:17:07.122960 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:17:07.122941    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3\": container with ID starting with aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3 not found: ID does not exist" containerID="aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3"
Sep 16 19:17:07.123015 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:07.122969    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3"} err="failed to get container status \"aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3\": rpc error: code = NotFound desc = could not find container \"aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3\": container with ID starting with aff7d688db6c6fa5b78286fb9ed38ad295cfc3d249bd8f894b9189d4002847d3 not found: ID does not exist"
Sep 16 19:17:07.134908 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:07.134889    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"]
Sep 16 19:17:07.142310 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:07.142288    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69j2ddd"]
Sep 16 19:17:08.703786 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:08.703753    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b96fa0ac-f519-4458-84a6-ae8e24c04843" path="/var/lib/kubelet/pods/b96fa0ac-f519-4458-84a6-ae8e24c04843/volumes"
Sep 16 19:17:10.526000 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:10.525957    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:11.494391 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:11.494340    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:11.594748 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:11.594716    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:17.819170 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:17.819126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:20.595507 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.595442    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc" podUID="98ed0bf3-e77e-493f-bb59-32fba37d1b53" containerName="main" containerID="cri-o://9303b989cd6ebdd5a637c6e3736182503581630f1da5604f066cb842e4d3ff09" gracePeriod=45
Sep 16 19:17:20.861780 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.861712    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:17:20.879835 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.879808    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn47h\" (UniqueName: \"kubernetes.io/projected/98ed0bf3-e77e-493f-bb59-32fba37d1b53-kube-api-access-gn47h\") pod \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") "
Sep 16 19:17:20.879936 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.879869    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-home\") pod \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") "
Sep 16 19:17:20.879936 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.879896    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ed0bf3-e77e-493f-bb59-32fba37d1b53-tls-certs\") pod \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") "
Sep 16 19:17:20.879936 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.879923    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-tmp-dir\") pod \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") "
Sep 16 19:17:20.880076 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.879972    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-model-cache\") pod \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") "
Sep 16 19:17:20.880076 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.880001    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-dshm\") pod \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\" (UID: \"98ed0bf3-e77e-493f-bb59-32fba37d1b53\") "
Sep 16 19:17:20.880187 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.880103    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-home" (OuterVolumeSpecName: "home") pod "98ed0bf3-e77e-493f-bb59-32fba37d1b53" (UID: "98ed0bf3-e77e-493f-bb59-32fba37d1b53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:20.880272 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.880254    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:17:20.880324 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.880250    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-model-cache" (OuterVolumeSpecName: "model-cache") pod "98ed0bf3-e77e-493f-bb59-32fba37d1b53" (UID: "98ed0bf3-e77e-493f-bb59-32fba37d1b53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:20.880324 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.880260    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98ed0bf3-e77e-493f-bb59-32fba37d1b53" (UID: "98ed0bf3-e77e-493f-bb59-32fba37d1b53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:20.882488 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.882445    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-dshm" (OuterVolumeSpecName: "dshm") pod "98ed0bf3-e77e-493f-bb59-32fba37d1b53" (UID: "98ed0bf3-e77e-493f-bb59-32fba37d1b53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:20.883183 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.882747    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ed0bf3-e77e-493f-bb59-32fba37d1b53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98ed0bf3-e77e-493f-bb59-32fba37d1b53" (UID: "98ed0bf3-e77e-493f-bb59-32fba37d1b53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:20.883183 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.882882    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ed0bf3-e77e-493f-bb59-32fba37d1b53-kube-api-access-gn47h" (OuterVolumeSpecName: "kube-api-access-gn47h") pod "98ed0bf3-e77e-493f-bb59-32fba37d1b53" (UID: "98ed0bf3-e77e-493f-bb59-32fba37d1b53"). InnerVolumeSpecName "kube-api-access-gn47h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:20.980783 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.980745    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:17:20.980783 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.980787    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn47h\" (UniqueName: \"kubernetes.io/projected/98ed0bf3-e77e-493f-bb59-32fba37d1b53-kube-api-access-gn47h\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:17:20.980992 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.980807    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ed0bf3-e77e-493f-bb59-32fba37d1b53-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:17:20.980992 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.980824    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:17:20.980992 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:20.980838    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ed0bf3-e77e-493f-bb59-32fba37d1b53-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:17:21.184805 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.184699    2582 generic.go:356] "Generic (PLEG): container finished" podID="98ed0bf3-e77e-493f-bb59-32fba37d1b53" containerID="9303b989cd6ebdd5a637c6e3736182503581630f1da5604f066cb842e4d3ff09" exitCode=0
Sep 16 19:17:21.184805 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.184767    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"
Sep 16 19:17:21.184805 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.184788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc" event={"ID":"98ed0bf3-e77e-493f-bb59-32fba37d1b53","Type":"ContainerDied","Data":"9303b989cd6ebdd5a637c6e3736182503581630f1da5604f066cb842e4d3ff09"}
Sep 16 19:17:21.185038 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.184874    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc" event={"ID":"98ed0bf3-e77e-493f-bb59-32fba37d1b53","Type":"ContainerDied","Data":"3a62ee6f28e6271be1ee6908cd0a6b125f7e3c419a725479fa9edd8c8d89a6ee"}
Sep 16 19:17:21.185038 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.184898    2582 scope.go:117] "RemoveContainer" containerID="9303b989cd6ebdd5a637c6e3736182503581630f1da5604f066cb842e4d3ff09"
Sep 16 19:17:21.195750 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.195723    2582 scope.go:117] "RemoveContainer" containerID="9303b989cd6ebdd5a637c6e3736182503581630f1da5604f066cb842e4d3ff09"
Sep 16 19:17:21.196105 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:17:21.196086    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9303b989cd6ebdd5a637c6e3736182503581630f1da5604f066cb842e4d3ff09\": container with ID starting with 9303b989cd6ebdd5a637c6e3736182503581630f1da5604f066cb842e4d3ff09 not found: ID does not exist" containerID="9303b989cd6ebdd5a637c6e3736182503581630f1da5604f066cb842e4d3ff09"
Sep 16 19:17:21.196167 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.196115    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9303b989cd6ebdd5a637c6e3736182503581630f1da5604f066cb842e4d3ff09"} err="failed to get container status \"9303b989cd6ebdd5a637c6e3736182503581630f1da5604f066cb842e4d3ff09\": rpc error: code = NotFound desc = could not find container \"9303b989cd6ebdd5a637c6e3736182503581630f1da5604f066cb842e4d3ff09\": container with ID starting with 9303b989cd6ebdd5a637c6e3736182503581630f1da5604f066cb842e4d3ff09 not found: ID does not exist"
Sep 16 19:17:21.207237 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.207199    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"]
Sep 16 19:17:21.207634 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.207616    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96fa0ac-f519-4458-84a6-ae8e24c04843" containerName="main"
Sep 16 19:17:21.207733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.207636    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96fa0ac-f519-4458-84a6-ae8e24c04843" containerName="main"
Sep 16 19:17:21.207733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.207655    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c0b8ad-8303-4f96-8b99-54750cbc13c1" containerName="main"
Sep 16 19:17:21.207733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.207664    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c0b8ad-8303-4f96-8b99-54750cbc13c1" containerName="main"
Sep 16 19:17:21.207733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.207689    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98ed0bf3-e77e-493f-bb59-32fba37d1b53" containerName="main"
Sep 16 19:17:21.207733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.207698    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="98ed0bf3-e77e-493f-bb59-32fba37d1b53" containerName="main"
Sep 16 19:17:21.207733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.207712    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c0b8ad-8303-4f96-8b99-54750cbc13c1" containerName="storage-initializer"
Sep 16 19:17:21.207733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.207720    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c0b8ad-8303-4f96-8b99-54750cbc13c1" containerName="storage-initializer"
Sep 16 19:17:21.208068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.207814    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b96fa0ac-f519-4458-84a6-ae8e24c04843" containerName="main"
Sep 16 19:17:21.208068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.207831    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="98ed0bf3-e77e-493f-bb59-32fba37d1b53" containerName="main"
Sep 16 19:17:21.208068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.207841    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c0b8ad-8303-4f96-8b99-54750cbc13c1" containerName="main"
Sep 16 19:17:21.212262 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.212228    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.213114 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.213088    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"]
Sep 16 19:17:21.215206 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.215184    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 16 19:17:21.215355 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.215185    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-skkvb\""
Sep 16 19:17:21.215355 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.215271    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 16 19:17:21.216760 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.216723    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 19:17:21.218602 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.218551    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xrngc"]
Sep 16 19:17:21.222140 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.222113    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"]
Sep 16 19:17:21.240684 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.240656    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:21.282149 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.282105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.282149 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.282152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-tls-certs\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.282363 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.282216    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htllp\" (UniqueName: \"kubernetes.io/projected/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-kube-api-access-htllp\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.282363 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.282254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-dshm\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.282363 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.282350    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-model-cache\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.282492 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.282369    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-home\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.383323 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.383281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.383323 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.383330    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-tls-certs\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.383621 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.383356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htllp\" (UniqueName: \"kubernetes.io/projected/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-kube-api-access-htllp\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.383621 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.383376    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-dshm\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.383621 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.383434    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-model-cache\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.383621 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.383558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-home\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.383848 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.383829    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.383894 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.383855    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-model-cache\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.383989 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.383970    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-home\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.385959 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.385942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-dshm\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.386236 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.386219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-tls-certs\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.393974 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.393942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htllp\" (UniqueName: \"kubernetes.io/projected/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-kube-api-access-htllp\") pod \"test-div-v1-kserve-fb566d667-swcr9\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.525248 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.525203    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:21.813998 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:21.813891    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"]
Sep 16 19:17:21.816605 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:17:21.816557    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8516f6_b1f0_4a2c_94d9_1c581cb6cd1b.slice/crio-9f645985d63e0eebbf87fe1983e59d290b0cb5f59e5ab61ed608148f87c4840d WatchSource:0}: Error finding container 9f645985d63e0eebbf87fe1983e59d290b0cb5f59e5ab61ed608148f87c4840d: Status 404 returned error can't find the container with id 9f645985d63e0eebbf87fe1983e59d290b0cb5f59e5ab61ed608148f87c4840d
Sep 16 19:17:22.193363 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:22.193263    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9" event={"ID":"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b","Type":"ContainerStarted","Data":"7d87d50466b6fcc9f0f6cdd356b9ec81be4f97f38acfea4d811041b31d772f03"}
Sep 16 19:17:22.193363 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:22.193315    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9" event={"ID":"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b","Type":"ContainerStarted","Data":"9f645985d63e0eebbf87fe1983e59d290b0cb5f59e5ab61ed608148f87c4840d"}
Sep 16 19:17:22.212411 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:22.212315    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9" podStartSLOduration=1.212297453 podStartE2EDuration="1.212297453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:22.211149909 +0000 UTC m=+1900.041758454" watchObservedRunningTime="2026-09-16 19:17:22.212297453 +0000 UTC m=+1900.042905997"
Sep 16 19:17:22.525622 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:22.525561    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:22.705145 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:22.705108    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98ed0bf3-e77e-493f-bb59-32fba37d1b53" path="/var/lib/kubelet/pods/98ed0bf3-e77e-493f-bb59-32fba37d1b53/volumes"
Sep 16 19:17:23.695434 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:23.695395    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:23.795377 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:23.795342    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:30.299734 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:30.299703    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:31.526349 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:31.526318    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:31.538312 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:31.538289    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:32.249752 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:32.249720    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:17:35.070215 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:35.070179    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:42.198097 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.198065    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"]
Sep 16 19:17:42.202269 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.202244    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.205145 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.205116    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 19:17:42.205634 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.205606    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:17:42.205763 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.205646    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-z7g2v\""
Sep 16 19:17:42.206512 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.206473    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 19:17:42.206661 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.206587    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 19:17:42.207036 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.207021    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 19:17:42.214455 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.214432    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"]
Sep 16 19:17:42.228048 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.228021    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:42.256240 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.256199    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-model-cache\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.256485 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.256251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff0293b-f68e-43a4-b0e1-483538bcd188-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.256485 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.256334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fff0293b-f68e-43a4-b0e1-483538bcd188-tls-certs\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.256485 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.256379    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7z62\" (UniqueName: \"kubernetes.io/projected/fff0293b-f68e-43a4-b0e1-483538bcd188-kube-api-access-n7z62\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.256485 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.256459    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.256722 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.256502    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-dshm\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.256722 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.256555    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.256722 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.256619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-home\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.357772 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.357734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.357986 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.357800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-home\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.357986 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.357837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-model-cache\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.357986 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.357874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff0293b-f68e-43a4-b0e1-483538bcd188-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.358127 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.358074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fff0293b-f68e-43a4-b0e1-483538bcd188-tls-certs\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.358181 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.358144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7z62\" (UniqueName: \"kubernetes.io/projected/fff0293b-f68e-43a4-b0e1-483538bcd188-kube-api-access-n7z62\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.358241 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.358209    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.358241 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.358227    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.358338 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.358253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-dshm\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.358338 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.358266    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-model-cache\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.358338 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.358316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-home\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.358497 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.358486    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.358747 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.358727    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff0293b-f68e-43a4-b0e1-483538bcd188-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.360648 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.360625    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-dshm\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.360778 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.360763    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fff0293b-f68e-43a4-b0e1-483538bcd188-tls-certs\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.366820 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.366794    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7z62\" (UniqueName: \"kubernetes.io/projected/fff0293b-f68e-43a4-b0e1-483538bcd188-kube-api-access-n7z62\") pod \"tokenizer-migration-path-test-kserve-56579b9954-cwph2\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.496292 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.496195    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"]
Sep 16 19:17:42.500945 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.500919    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"
Sep 16 19:17:42.504103 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.504063    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-cztl2\""
Sep 16 19:17:42.513538 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.513497    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"]
Sep 16 19:17:42.517729 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.517698    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:42.559708 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.559669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46sz5\" (UniqueName: \"kubernetes.io/projected/c25356ee-29fe-4cdb-9dad-2197b9fa5837-kube-api-access-46sz5\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wx654\" (UID: \"c25356ee-29fe-4cdb-9dad-2197b9fa5837\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"
Sep 16 19:17:42.559866 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.559735    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25356ee-29fe-4cdb-9dad-2197b9fa5837-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wx654\" (UID: \"c25356ee-29fe-4cdb-9dad-2197b9fa5837\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"
Sep 16 19:17:42.588500 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.588431    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:42.649842 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.649796    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:42.660990 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.660961    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46sz5\" (UniqueName: \"kubernetes.io/projected/c25356ee-29fe-4cdb-9dad-2197b9fa5837-kube-api-access-46sz5\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wx654\" (UID: \"c25356ee-29fe-4cdb-9dad-2197b9fa5837\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"
Sep 16 19:17:42.661093 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.661026    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25356ee-29fe-4cdb-9dad-2197b9fa5837-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wx654\" (UID: \"c25356ee-29fe-4cdb-9dad-2197b9fa5837\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"
Sep 16 19:17:42.663855 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.663827    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25356ee-29fe-4cdb-9dad-2197b9fa5837-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wx654\" (UID: \"c25356ee-29fe-4cdb-9dad-2197b9fa5837\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"
Sep 16 19:17:42.672218 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.672190    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46sz5\" (UniqueName: \"kubernetes.io/projected/c25356ee-29fe-4cdb-9dad-2197b9fa5837-kube-api-access-46sz5\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wx654\" (UID: \"c25356ee-29fe-4cdb-9dad-2197b9fa5837\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"
Sep 16 19:17:42.799305 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.799257    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"]
Sep 16 19:17:42.801258 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:17:42.801229    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff0293b_f68e_43a4_b0e1_483538bcd188.slice/crio-bfab6757a7f4a9fc4077d7742aaf735c03ad1ca97de2214ab64d4b9e648f06e4 WatchSource:0}: Error finding container bfab6757a7f4a9fc4077d7742aaf735c03ad1ca97de2214ab64d4b9e648f06e4: Status 404 returned error can't find the container with id bfab6757a7f4a9fc4077d7742aaf735c03ad1ca97de2214ab64d4b9e648f06e4
Sep 16 19:17:42.806248 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.806227    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 19:17:42.820767 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.820746    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-cztl2\""
Sep 16 19:17:42.828565 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:42.828538    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"
Sep 16 19:17:43.108398 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:43.108365    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"]
Sep 16 19:17:43.110116 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:17:43.110088    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25356ee_29fe_4cdb_9dad_2197b9fa5837.slice/crio-77e0713db1992b574736382db92d146fb379bf9f2322cd00b0d8abaa961958b2 WatchSource:0}: Error finding container 77e0713db1992b574736382db92d146fb379bf9f2322cd00b0d8abaa961958b2: Status 404 returned error can't find the container with id 77e0713db1992b574736382db92d146fb379bf9f2322cd00b0d8abaa961958b2
Sep 16 19:17:43.296619 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:43.296551    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2" event={"ID":"fff0293b-f68e-43a4-b0e1-483538bcd188","Type":"ContainerStarted","Data":"2ac21b3818af539ce3af9bc17d9fbbb80bf7183e04395b07619b49c73cbc666b"}
Sep 16 19:17:43.297121 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:43.296646    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2" event={"ID":"fff0293b-f68e-43a4-b0e1-483538bcd188","Type":"ContainerStarted","Data":"bfab6757a7f4a9fc4077d7742aaf735c03ad1ca97de2214ab64d4b9e648f06e4"}
Sep 16 19:17:43.298489 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:43.298460    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654" event={"ID":"c25356ee-29fe-4cdb-9dad-2197b9fa5837","Type":"ContainerStarted","Data":"09175ac30f061ef4e9182bd5bb2fa429f3c1358bc1af8ec0ad6f4cd20c96a7e9"}
Sep 16 19:17:43.298648 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:43.298509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654" event={"ID":"c25356ee-29fe-4cdb-9dad-2197b9fa5837","Type":"ContainerStarted","Data":"77e0713db1992b574736382db92d146fb379bf9f2322cd00b0d8abaa961958b2"}
Sep 16 19:17:43.298648 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:43.298531    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"
Sep 16 19:17:43.343487 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:43.343412    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654" podStartSLOduration=1.3433851780000001 podStartE2EDuration="1.343385178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:43.341544756 +0000 UTC m=+1921.172153331" watchObservedRunningTime="2026-09-16 19:17:43.343385178 +0000 UTC m=+1921.173993724"
Sep 16 19:17:46.317845 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:46.317808    2582 generic.go:356] "Generic (PLEG): container finished" podID="fff0293b-f68e-43a4-b0e1-483538bcd188" containerID="2ac21b3818af539ce3af9bc17d9fbbb80bf7183e04395b07619b49c73cbc666b" exitCode=0
Sep 16 19:17:46.318247 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:46.317849    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2" event={"ID":"fff0293b-f68e-43a4-b0e1-483538bcd188","Type":"ContainerDied","Data":"2ac21b3818af539ce3af9bc17d9fbbb80bf7183e04395b07619b49c73cbc666b"}
Sep 16 19:17:47.324770 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:47.324729    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2" event={"ID":"fff0293b-f68e-43a4-b0e1-483538bcd188","Type":"ContainerStarted","Data":"c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3"}
Sep 16 19:17:47.352562 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:47.352501    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2" podStartSLOduration=2.850283513 podStartE2EDuration="5.352483527s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.502200014s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:47.349736444 +0000 UTC m=+1925.180344990" watchObservedRunningTime="2026-09-16 19:17:47.352483527 +0000 UTC m=+1925.183092071"
Sep 16 19:17:47.518064 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:47.518017    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:47.849677 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:47.849630    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:52.519099 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:52.519052    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:52.531742 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:52.531709    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:53.363006 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:53.362975    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:17:53.484932 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:53.484898    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:53.585073 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:17:53.585033    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:11.484476 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:11.484443    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:13.627699 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:13.627666    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:13.728261 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:13.728206    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:13.870056 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:13.870024    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:14.307834 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:14.307803    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"
Sep 16 19:18:15.197930 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:15.197891    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"]
Sep 16 19:18:15.218637 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:15.218597    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:16.107949 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:16.107919    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:18.707461 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:18.707424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:22.537309 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:22.537268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:25.300250 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.300198    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"]
Sep 16 19:18:25.303747 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.303724    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.308262 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.308229    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 19:18:25.308434 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.308229    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-ggb2p\""
Sep 16 19:18:25.308434 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.308230    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 16 19:18:25.308434 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.308245    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 16 19:18:25.315531 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.315503    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"]
Sep 16 19:18:25.356169 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.356127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.356390 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.356206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.356390 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.356355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl6df\" (UniqueName: \"kubernetes.io/projected/cfa5bc6a-533e-4917-b2e3-47d0200fc172-kube-api-access-kl6df\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.356490 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.356465    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa5bc6a-533e-4917-b2e3-47d0200fc172-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.356528 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.356497    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-home\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.356607 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.356541    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.389830 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.389795    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:25.457375 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.457332    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.457375 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.457383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.457722 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.457442    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl6df\" (UniqueName: \"kubernetes.io/projected/cfa5bc6a-533e-4917-b2e3-47d0200fc172-kube-api-access-kl6df\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.457722 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.457513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa5bc6a-533e-4917-b2e3-47d0200fc172-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.457722 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.457541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-home\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.457722 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.457607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.457970 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.457943    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.458027 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.457978    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.458071 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.458030    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-home\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.460204 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.460180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.460427 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.460408    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa5bc6a-533e-4917-b2e3-47d0200fc172-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.467538 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.467512    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl6df\" (UniqueName: \"kubernetes.io/projected/cfa5bc6a-533e-4917-b2e3-47d0200fc172-kube-api-access-kl6df\") pod \"test-div-v2-kserve-5f6fdb6b77-zr7dw\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.616156 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.616033    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:25.898526 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:25.898494    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"]
Sep 16 19:18:25.901541 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:18:25.901505    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa5bc6a_533e_4917_b2e3_47d0200fc172.slice/crio-6c1a1113e0e382f1ab4d38e490ccf379a8cf8e3c5ca04695cde38b3eb14e31ed WatchSource:0}: Error finding container 6c1a1113e0e382f1ab4d38e490ccf379a8cf8e3c5ca04695cde38b3eb14e31ed: Status 404 returned error can't find the container with id 6c1a1113e0e382f1ab4d38e490ccf379a8cf8e3c5ca04695cde38b3eb14e31ed
Sep 16 19:18:26.516941 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:26.516895    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw" event={"ID":"cfa5bc6a-533e-4917-b2e3-47d0200fc172","Type":"ContainerStarted","Data":"9f47d4523d3f7771f9b9774dd1001daf0ed7b61d1bbd31d7fb2af832e71e799d"}
Sep 16 19:18:26.517408 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:26.516970    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw" event={"ID":"cfa5bc6a-533e-4917-b2e3-47d0200fc172","Type":"ContainerStarted","Data":"6c1a1113e0e382f1ab4d38e490ccf379a8cf8e3c5ca04695cde38b3eb14e31ed"}
Sep 16 19:18:26.536790 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:26.536727    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw" podStartSLOduration=1.5367064639999999 podStartE2EDuration="1.536706464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:26.535322749 +0000 UTC m=+1964.365931295" watchObservedRunningTime="2026-09-16 19:18:26.536706464 +0000 UTC m=+1964.367315009"
Sep 16 19:18:26.616258 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:26.616208    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:30.218652 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.218568    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9" podUID="2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b" containerName="main" containerID="cri-o://7d87d50466b6fcc9f0f6cdd356b9ec81be4f97f38acfea4d811041b31d772f03" gracePeriod=45
Sep 16 19:18:30.239454 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.239419    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9" podUID="2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 16 19:18:30.502124 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.502088    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:18:30.542756 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.542704    2582 generic.go:356] "Generic (PLEG): container finished" podID="2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b" containerID="7d87d50466b6fcc9f0f6cdd356b9ec81be4f97f38acfea4d811041b31d772f03" exitCode=0
Sep 16 19:18:30.543149 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.542767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9" event={"ID":"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b","Type":"ContainerDied","Data":"7d87d50466b6fcc9f0f6cdd356b9ec81be4f97f38acfea4d811041b31d772f03"}
Sep 16 19:18:30.543149 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.542780    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"
Sep 16 19:18:30.543149 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.542842    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9" event={"ID":"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b","Type":"ContainerDied","Data":"9f645985d63e0eebbf87fe1983e59d290b0cb5f59e5ab61ed608148f87c4840d"}
Sep 16 19:18:30.543149 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.542877    2582 scope.go:117] "RemoveContainer" containerID="7d87d50466b6fcc9f0f6cdd356b9ec81be4f97f38acfea4d811041b31d772f03"
Sep 16 19:18:30.554602 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.554550    2582 scope.go:117] "RemoveContainer" containerID="7d87d50466b6fcc9f0f6cdd356b9ec81be4f97f38acfea4d811041b31d772f03"
Sep 16 19:18:30.555040 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:18:30.555017    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d87d50466b6fcc9f0f6cdd356b9ec81be4f97f38acfea4d811041b31d772f03\": container with ID starting with 7d87d50466b6fcc9f0f6cdd356b9ec81be4f97f38acfea4d811041b31d772f03 not found: ID does not exist" containerID="7d87d50466b6fcc9f0f6cdd356b9ec81be4f97f38acfea4d811041b31d772f03"
Sep 16 19:18:30.555139 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.555052    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d87d50466b6fcc9f0f6cdd356b9ec81be4f97f38acfea4d811041b31d772f03"} err="failed to get container status \"7d87d50466b6fcc9f0f6cdd356b9ec81be4f97f38acfea4d811041b31d772f03\": rpc error: code = NotFound desc = could not find container \"7d87d50466b6fcc9f0f6cdd356b9ec81be4f97f38acfea4d811041b31d772f03\": container with ID starting with 7d87d50466b6fcc9f0f6cdd356b9ec81be4f97f38acfea4d811041b31d772f03 not found: ID does not exist"
Sep 16 19:18:30.609725 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.609662    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-dshm\") pod \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") "
Sep 16 19:18:30.609979 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.609824    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-tmp-dir\") pod \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") "
Sep 16 19:18:30.609979 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.609862    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-home\") pod \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") "
Sep 16 19:18:30.609979 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.609912    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htllp\" (UniqueName: \"kubernetes.io/projected/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-kube-api-access-htllp\") pod \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") "
Sep 16 19:18:30.609979 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.609974    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-tls-certs\") pod \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") "
Sep 16 19:18:30.610196 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.610005    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-model-cache\") pod \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\" (UID: \"2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b\") "
Sep 16 19:18:30.610255 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.610189    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b" (UID: "2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:30.610255 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.610217    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-home" (OuterVolumeSpecName: "home") pod "2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b" (UID: "2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:30.610427 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.610390    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b" (UID: "2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:30.610505 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.610409    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:18:30.610505 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.610459    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:18:30.612828 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.612798    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-dshm" (OuterVolumeSpecName: "dshm") pod "2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b" (UID: "2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:30.612985 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.612846    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-kube-api-access-htllp" (OuterVolumeSpecName: "kube-api-access-htllp") pod "2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b" (UID: "2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b"). InnerVolumeSpecName "kube-api-access-htllp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:30.612985 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.612844    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b" (UID: "2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:30.711303 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.711250    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:18:30.711303 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.711285    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:18:30.711303 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.711311    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:18:30.711735 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.711323    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htllp\" (UniqueName: \"kubernetes.io/projected/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b-kube-api-access-htllp\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:18:30.861237 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.861186    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"]
Sep 16 19:18:30.867603 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:30.867554    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-swcr9"]
Sep 16 19:18:32.705396 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:32.705354    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b" path="/var/lib/kubelet/pods/2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b/volumes"
Sep 16 19:18:35.617316 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:35.617199    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:35.630755 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:35.630720    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:36.588983 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:36.588947    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:18:55.640469 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:55.640422    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:55.664825 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:55.664773    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"]
Sep 16 19:18:55.690421 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:55.690346    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"]
Sep 16 19:18:55.701256 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:18:55.701209    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:18:55.701256 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:18:55.701256 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:18:55.701256 ip-10-0-142-173 kubenswrapper[2582]:  > containerID="c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3" cmd=["/bin/sleep","15"]
Sep 16 19:18:55.701599 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:55.701278    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2" message="" err=<
Sep 16 19:18:55.701599 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:18:55.701599 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:18:55.701599 ip-10-0-142-173 kubenswrapper[2582]:  >
Sep 16 19:18:55.701599 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:18:55.701293    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:18:55.701599 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:18:55.701599 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:18:55.701599 ip-10-0-142-173 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2" podUID="fff0293b-f68e-43a4-b0e1-483538bcd188" containerName="main" containerID="cri-o://c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3"
Sep 16 19:18:55.701599 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:55.701357    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2" podUID="fff0293b-f68e-43a4-b0e1-483538bcd188" containerName="main" containerID="cri-o://c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3" gracePeriod=60
Sep 16 19:18:55.736409 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:55.736358    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:56.031196 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.031162    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:56.352110 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.352062    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2" podUID="fff0293b-f68e-43a4-b0e1-483538bcd188" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 16 19:18:56.469266 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.469227    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:18:56.562940 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.562893    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff0293b-f68e-43a4-b0e1-483538bcd188-cabundle-cert\") pod \"fff0293b-f68e-43a4-b0e1-483538bcd188\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") "
Sep 16 19:18:56.563207 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.562970    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-kserve-provision-location\") pod \"fff0293b-f68e-43a4-b0e1-483538bcd188\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") "
Sep 16 19:18:56.563207 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.562999    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-home\") pod \"fff0293b-f68e-43a4-b0e1-483538bcd188\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") "
Sep 16 19:18:56.563207 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.563050    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fff0293b-f68e-43a4-b0e1-483538bcd188-tls-certs\") pod \"fff0293b-f68e-43a4-b0e1-483538bcd188\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") "
Sep 16 19:18:56.563207 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.563073    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7z62\" (UniqueName: \"kubernetes.io/projected/fff0293b-f68e-43a4-b0e1-483538bcd188-kube-api-access-n7z62\") pod \"fff0293b-f68e-43a4-b0e1-483538bcd188\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") "
Sep 16 19:18:56.563207 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.563191    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-model-cache\") pod \"fff0293b-f68e-43a4-b0e1-483538bcd188\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") "
Sep 16 19:18:56.563518 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.563213    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-dshm\") pod \"fff0293b-f68e-43a4-b0e1-483538bcd188\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") "
Sep 16 19:18:56.563518 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.563258    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-tmp-dir\") pod \"fff0293b-f68e-43a4-b0e1-483538bcd188\" (UID: \"fff0293b-f68e-43a4-b0e1-483538bcd188\") "
Sep 16 19:18:56.563518 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.563393    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff0293b-f68e-43a4-b0e1-483538bcd188-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fff0293b-f68e-43a4-b0e1-483538bcd188" (UID: "fff0293b-f68e-43a4-b0e1-483538bcd188"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:56.563518 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.563390    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-home" (OuterVolumeSpecName: "home") pod "fff0293b-f68e-43a4-b0e1-483538bcd188" (UID: "fff0293b-f68e-43a4-b0e1-483538bcd188"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.563518 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.563462    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-model-cache" (OuterVolumeSpecName: "model-cache") pod "fff0293b-f68e-43a4-b0e1-483538bcd188" (UID: "fff0293b-f68e-43a4-b0e1-483538bcd188"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.563795 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.563665    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fff0293b-f68e-43a4-b0e1-483538bcd188" (UID: "fff0293b-f68e-43a4-b0e1-483538bcd188"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.563795 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.563705    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.563795 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.563721    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff0293b-f68e-43a4-b0e1-483538bcd188-cabundle-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.563795 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.563734    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.565967 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.565934    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff0293b-f68e-43a4-b0e1-483538bcd188-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fff0293b-f68e-43a4-b0e1-483538bcd188" (UID: "fff0293b-f68e-43a4-b0e1-483538bcd188"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:56.566184 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.566164    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-dshm" (OuterVolumeSpecName: "dshm") pod "fff0293b-f68e-43a4-b0e1-483538bcd188" (UID: "fff0293b-f68e-43a4-b0e1-483538bcd188"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.566269 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.566194    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff0293b-f68e-43a4-b0e1-483538bcd188-kube-api-access-n7z62" (OuterVolumeSpecName: "kube-api-access-n7z62") pod "fff0293b-f68e-43a4-b0e1-483538bcd188" (UID: "fff0293b-f68e-43a4-b0e1-483538bcd188"). InnerVolumeSpecName "kube-api-access-n7z62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:56.586657 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.586618    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fff0293b-f68e-43a4-b0e1-483538bcd188" (UID: "fff0293b-f68e-43a4-b0e1-483538bcd188"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.664362 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.664315    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.664362 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.664354    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.664362 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.664365    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff0293b-f68e-43a4-b0e1-483538bcd188-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.664362 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.664375    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fff0293b-f68e-43a4-b0e1-483538bcd188-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.664952 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.664388    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7z62\" (UniqueName: \"kubernetes.io/projected/fff0293b-f68e-43a4-b0e1-483538bcd188-kube-api-access-n7z62\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.682621 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.682556    2582 generic.go:356] "Generic (PLEG): container finished" podID="fff0293b-f68e-43a4-b0e1-483538bcd188" containerID="c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3" exitCode=0
Sep 16 19:18:56.682761 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.682641    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"
Sep 16 19:18:56.682761 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.682649    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2" event={"ID":"fff0293b-f68e-43a4-b0e1-483538bcd188","Type":"ContainerDied","Data":"c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3"}
Sep 16 19:18:56.682761 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.682720    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2" event={"ID":"fff0293b-f68e-43a4-b0e1-483538bcd188","Type":"ContainerDied","Data":"bfab6757a7f4a9fc4077d7742aaf735c03ad1ca97de2214ab64d4b9e648f06e4"}
Sep 16 19:18:56.682761 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.682746    2582 scope.go:117] "RemoveContainer" containerID="c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3"
Sep 16 19:18:56.695569 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.695539    2582 scope.go:117] "RemoveContainer" containerID="2ac21b3818af539ce3af9bc17d9fbbb80bf7183e04395b07619b49c73cbc666b"
Sep 16 19:18:56.706876 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.706619    2582 scope.go:117] "RemoveContainer" containerID="c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3"
Sep 16 19:18:56.706961 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:18:56.706939    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3\": container with ID starting with c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3 not found: ID does not exist" containerID="c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3"
Sep 16 19:18:56.707008 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.706974    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3"} err="failed to get container status \"c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3\": rpc error: code = NotFound desc = could not find container \"c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3\": container with ID starting with c37bcc1bf1194d69c4bbfa7e347983f11ccb00f2bf5f41e452a0de5b177190f3 not found: ID does not exist"
Sep 16 19:18:56.707008 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.706994    2582 scope.go:117] "RemoveContainer" containerID="2ac21b3818af539ce3af9bc17d9fbbb80bf7183e04395b07619b49c73cbc666b"
Sep 16 19:18:56.707257 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:18:56.707240    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ac21b3818af539ce3af9bc17d9fbbb80bf7183e04395b07619b49c73cbc666b\": container with ID starting with 2ac21b3818af539ce3af9bc17d9fbbb80bf7183e04395b07619b49c73cbc666b not found: ID does not exist" containerID="2ac21b3818af539ce3af9bc17d9fbbb80bf7183e04395b07619b49c73cbc666b"
Sep 16 19:18:56.707344 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.707260    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ac21b3818af539ce3af9bc17d9fbbb80bf7183e04395b07619b49c73cbc666b"} err="failed to get container status \"2ac21b3818af539ce3af9bc17d9fbbb80bf7183e04395b07619b49c73cbc666b\": rpc error: code = NotFound desc = could not find container \"2ac21b3818af539ce3af9bc17d9fbbb80bf7183e04395b07619b49c73cbc666b\": container with ID starting with 2ac21b3818af539ce3af9bc17d9fbbb80bf7183e04395b07619b49c73cbc666b not found: ID does not exist"
Sep 16 19:18:56.710479 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.710454    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"]
Sep 16 19:18:56.716066 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:56.716030    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-cwph2"]
Sep 16 19:18:58.703920 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:18:58.703872    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fff0293b-f68e-43a4-b0e1-483538bcd188" path="/var/lib/kubelet/pods/fff0293b-f68e-43a4-b0e1-483538bcd188/volumes"
Sep 16 19:19:01.818426 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:01.818355    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:01.916926 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:01.916878    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:03.211988 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:03.211945    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:03.315536 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:03.315499    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:03.414762 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:03.414732    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:03.641677 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:03.641638    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:03.731920 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:03.731881    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"]
Sep 16 19:19:03.775638 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:03.775563    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:03.870599 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:03.870542    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:04.162886 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:04.162840    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:04.308550 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:04.308510    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654" podUID="c25356ee-29fe-4cdb-9dad-2197b9fa5837" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:19:08.288427 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:08.288381    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:09.435133 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:09.435091    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:09.511589 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:09.511531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:10.689020 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:10.688970    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654" podUID="c25356ee-29fe-4cdb-9dad-2197b9fa5837" containerName="main" containerID="cri-o://09175ac30f061ef4e9182bd5bb2fa429f3c1358bc1af8ec0ad6f4cd20c96a7e9" gracePeriod=45
Sep 16 19:19:14.307768 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:14.307725    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654" podUID="c25356ee-29fe-4cdb-9dad-2197b9fa5837" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:19:15.116105 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:15.116064    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:15.192287 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:15.192240    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:18.753371 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:18.753322    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw" podUID="cfa5bc6a-533e-4917-b2e3-47d0200fc172" containerName="main" containerID="cri-o://9f47d4523d3f7771f9b9774dd1001daf0ed7b61d1bbd31d7fb2af832e71e799d" gracePeriod=45
Sep 16 19:19:19.010556 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.010473    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:19:19.190079 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.190034    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-home\") pod \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") "
Sep 16 19:19:19.190079 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.190077    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-dshm\") pod \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") "
Sep 16 19:19:19.190366 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.190111    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-model-cache\") pod \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") "
Sep 16 19:19:19.190366 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.190160    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-tmp-dir\") pod \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") "
Sep 16 19:19:19.190366 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.190180    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kl6df\" (UniqueName: \"kubernetes.io/projected/cfa5bc6a-533e-4917-b2e3-47d0200fc172-kube-api-access-kl6df\") pod \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") "
Sep 16 19:19:19.190366 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.190198    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa5bc6a-533e-4917-b2e3-47d0200fc172-tls-certs\") pod \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\" (UID: \"cfa5bc6a-533e-4917-b2e3-47d0200fc172\") "
Sep 16 19:19:19.190707 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.190446    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-home" (OuterVolumeSpecName: "home") pod "cfa5bc6a-533e-4917-b2e3-47d0200fc172" (UID: "cfa5bc6a-533e-4917-b2e3-47d0200fc172"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:19.190707 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.190470    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfa5bc6a-533e-4917-b2e3-47d0200fc172" (UID: "cfa5bc6a-533e-4917-b2e3-47d0200fc172"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:19.190707 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.190528    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfa5bc6a-533e-4917-b2e3-47d0200fc172" (UID: "cfa5bc6a-533e-4917-b2e3-47d0200fc172"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:19.193072 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.193038    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa5bc6a-533e-4917-b2e3-47d0200fc172-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfa5bc6a-533e-4917-b2e3-47d0200fc172" (UID: "cfa5bc6a-533e-4917-b2e3-47d0200fc172"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:19.193072 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.193054    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa5bc6a-533e-4917-b2e3-47d0200fc172-kube-api-access-kl6df" (OuterVolumeSpecName: "kube-api-access-kl6df") pod "cfa5bc6a-533e-4917-b2e3-47d0200fc172" (UID: "cfa5bc6a-533e-4917-b2e3-47d0200fc172"). InnerVolumeSpecName "kube-api-access-kl6df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:19.193266 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.193053    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-dshm" (OuterVolumeSpecName: "dshm") pod "cfa5bc6a-533e-4917-b2e3-47d0200fc172" (UID: "cfa5bc6a-533e-4917-b2e3-47d0200fc172"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:19.291636 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.291556    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:19:19.291636 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.291633    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl6df\" (UniqueName: \"kubernetes.io/projected/cfa5bc6a-533e-4917-b2e3-47d0200fc172-kube-api-access-kl6df\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:19:19.291636 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.291643    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa5bc6a-533e-4917-b2e3-47d0200fc172-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:19:19.291954 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.291654    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:19:19.291954 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.291662    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:19:19.291954 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.291671    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa5bc6a-533e-4917-b2e3-47d0200fc172-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:19:19.798589 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.798533    2582 generic.go:356] "Generic (PLEG): container finished" podID="cfa5bc6a-533e-4917-b2e3-47d0200fc172" containerID="9f47d4523d3f7771f9b9774dd1001daf0ed7b61d1bbd31d7fb2af832e71e799d" exitCode=0
Sep 16 19:19:19.799212 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.798614    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw" event={"ID":"cfa5bc6a-533e-4917-b2e3-47d0200fc172","Type":"ContainerDied","Data":"9f47d4523d3f7771f9b9774dd1001daf0ed7b61d1bbd31d7fb2af832e71e799d"}
Sep 16 19:19:19.799212 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.798637    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"
Sep 16 19:19:19.799212 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.798687    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw" event={"ID":"cfa5bc6a-533e-4917-b2e3-47d0200fc172","Type":"ContainerDied","Data":"6c1a1113e0e382f1ab4d38e490ccf379a8cf8e3c5ca04695cde38b3eb14e31ed"}
Sep 16 19:19:19.799212 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.798715    2582 scope.go:117] "RemoveContainer" containerID="9f47d4523d3f7771f9b9774dd1001daf0ed7b61d1bbd31d7fb2af832e71e799d"
Sep 16 19:19:19.809197 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.809176    2582 scope.go:117] "RemoveContainer" containerID="9f47d4523d3f7771f9b9774dd1001daf0ed7b61d1bbd31d7fb2af832e71e799d"
Sep 16 19:19:19.809507 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:19:19.809486    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f47d4523d3f7771f9b9774dd1001daf0ed7b61d1bbd31d7fb2af832e71e799d\": container with ID starting with 9f47d4523d3f7771f9b9774dd1001daf0ed7b61d1bbd31d7fb2af832e71e799d not found: ID does not exist" containerID="9f47d4523d3f7771f9b9774dd1001daf0ed7b61d1bbd31d7fb2af832e71e799d"
Sep 16 19:19:19.809568 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.809517    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f47d4523d3f7771f9b9774dd1001daf0ed7b61d1bbd31d7fb2af832e71e799d"} err="failed to get container status \"9f47d4523d3f7771f9b9774dd1001daf0ed7b61d1bbd31d7fb2af832e71e799d\": rpc error: code = NotFound desc = could not find container \"9f47d4523d3f7771f9b9774dd1001daf0ed7b61d1bbd31d7fb2af832e71e799d\": container with ID starting with 9f47d4523d3f7771f9b9774dd1001daf0ed7b61d1bbd31d7fb2af832e71e799d not found: ID does not exist"
Sep 16 19:19:19.822273 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.822237    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"]
Sep 16 19:19:19.823994 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:19.823971    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-zr7dw"]
Sep 16 19:19:20.704823 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:20.704774    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfa5bc6a-533e-4917-b2e3-47d0200fc172" path="/var/lib/kubelet/pods/cfa5bc6a-533e-4917-b2e3-47d0200fc172/volumes"
Sep 16 19:19:24.308500 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:24.308456    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654" podUID="c25356ee-29fe-4cdb-9dad-2197b9fa5837" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:19:24.308880 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:24.308617    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"
Sep 16 19:19:26.098543 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.098504    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"]
Sep 16 19:19:26.099126 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.099102    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff0293b-f68e-43a4-b0e1-483538bcd188" containerName="storage-initializer"
Sep 16 19:19:26.099171 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.099131    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff0293b-f68e-43a4-b0e1-483538bcd188" containerName="storage-initializer"
Sep 16 19:19:26.099171 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.099145    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa5bc6a-533e-4917-b2e3-47d0200fc172" containerName="main"
Sep 16 19:19:26.099171 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.099154    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa5bc6a-533e-4917-b2e3-47d0200fc172" containerName="main"
Sep 16 19:19:26.099171 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.099167    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b" containerName="main"
Sep 16 19:19:26.099320 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.099176    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b" containerName="main"
Sep 16 19:19:26.099320 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.099186    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff0293b-f68e-43a4-b0e1-483538bcd188" containerName="main"
Sep 16 19:19:26.099320 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.099192    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff0293b-f68e-43a4-b0e1-483538bcd188" containerName="main"
Sep 16 19:19:26.099320 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.099309    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa5bc6a-533e-4917-b2e3-47d0200fc172" containerName="main"
Sep 16 19:19:26.099446 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.099323    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a8516f6-b1f0-4a2c-94d9-1c581cb6cd1b" containerName="main"
Sep 16 19:19:26.099446 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.099338    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fff0293b-f68e-43a4-b0e1-483538bcd188" containerName="main"
Sep 16 19:19:26.103808 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.103784    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.108425 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.108395    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 16 19:19:26.108623 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.108431    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-62x4j\""
Sep 16 19:19:26.108623 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.108450    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 16 19:19:26.108623 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.108498    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 16 19:19:26.112865 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.112835    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"]
Sep 16 19:19:26.134106 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.134074    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:26.153723 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.153685    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.153837 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.153750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmkqv\" (UniqueName: \"kubernetes.io/projected/8aadd537-e93b-4bb6-ac12-38a078e1deb2-kube-api-access-kmkqv\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.153837 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.153799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-dshm\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.153916 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.153830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-model-cache\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.153916 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.153869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aadd537-e93b-4bb6-ac12-38a078e1deb2-tls-certs\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.153916 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.153892    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-home\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.255301 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.255235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.255625 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.255324    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmkqv\" (UniqueName: \"kubernetes.io/projected/8aadd537-e93b-4bb6-ac12-38a078e1deb2-kube-api-access-kmkqv\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.255625 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.255377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-dshm\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.255625 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.255426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-model-cache\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.255625 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.255474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aadd537-e93b-4bb6-ac12-38a078e1deb2-tls-certs\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.255625 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.255508    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-home\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.255950 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.255844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.255950 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.255913    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-model-cache\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.256062 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.256041    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-home\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.258469 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.258424    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-dshm\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.258753 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.258734    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aadd537-e93b-4bb6-ac12-38a078e1deb2-tls-certs\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.265384 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.265348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmkqv\" (UniqueName: \"kubernetes.io/projected/8aadd537-e93b-4bb6-ac12-38a078e1deb2-kube-api-access-kmkqv\") pod \"stop-v2-kserve-7f68747c45-46nq5\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.415983 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.415838    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:26.705422 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:19:26.705384    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aadd537_e93b_4bb6_ac12_38a078e1deb2.slice/crio-7ceee2da3c0f129f9c1b0dd246898e9844fad08eb1908f67d0c20abf22d7c41b WatchSource:0}: Error finding container 7ceee2da3c0f129f9c1b0dd246898e9844fad08eb1908f67d0c20abf22d7c41b: Status 404 returned error can't find the container with id 7ceee2da3c0f129f9c1b0dd246898e9844fad08eb1908f67d0c20abf22d7c41b
Sep 16 19:19:26.706873 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.706843    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"]
Sep 16 19:19:26.836012 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.835962    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5" event={"ID":"8aadd537-e93b-4bb6-ac12-38a078e1deb2","Type":"ContainerStarted","Data":"ac74b9dedeaaaadb162feb04413256328d36f040ccfb9388c0ecdfa046a51c58"}
Sep 16 19:19:26.836261 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.836039    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5" event={"ID":"8aadd537-e93b-4bb6-ac12-38a078e1deb2","Type":"ContainerStarted","Data":"7ceee2da3c0f129f9c1b0dd246898e9844fad08eb1908f67d0c20abf22d7c41b"}
Sep 16 19:19:26.856857 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:26.856796    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5" podStartSLOduration=0.856775931 podStartE2EDuration="856.775931ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:26.855752105 +0000 UTC m=+2024.686360642" watchObservedRunningTime="2026-09-16 19:19:26.856775931 +0000 UTC m=+2024.687384477"
Sep 16 19:19:27.416883 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:27.416836    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:34.308223 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:34.308179    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654" podUID="c25356ee-29fe-4cdb-9dad-2197b9fa5837" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:19:36.416766 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:36.416710    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:36.430117 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:36.430086    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:36.896837 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:36.896801    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:19:40.906229 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:40.906191    2582 generic.go:356] "Generic (PLEG): container finished" podID="c25356ee-29fe-4cdb-9dad-2197b9fa5837" containerID="09175ac30f061ef4e9182bd5bb2fa429f3c1358bc1af8ec0ad6f4cd20c96a7e9" exitCode=0
Sep 16 19:19:40.906719 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:40.906265    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654" event={"ID":"c25356ee-29fe-4cdb-9dad-2197b9fa5837","Type":"ContainerDied","Data":"09175ac30f061ef4e9182bd5bb2fa429f3c1358bc1af8ec0ad6f4cd20c96a7e9"}
Sep 16 19:19:40.950755 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:40.950676    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"
Sep 16 19:19:41.001052 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:41.001014    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25356ee-29fe-4cdb-9dad-2197b9fa5837-tls-certs\") pod \"c25356ee-29fe-4cdb-9dad-2197b9fa5837\" (UID: \"c25356ee-29fe-4cdb-9dad-2197b9fa5837\") "
Sep 16 19:19:41.001329 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:41.001134    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46sz5\" (UniqueName: \"kubernetes.io/projected/c25356ee-29fe-4cdb-9dad-2197b9fa5837-kube-api-access-46sz5\") pod \"c25356ee-29fe-4cdb-9dad-2197b9fa5837\" (UID: \"c25356ee-29fe-4cdb-9dad-2197b9fa5837\") "
Sep 16 19:19:41.003751 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:41.003714    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25356ee-29fe-4cdb-9dad-2197b9fa5837-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c25356ee-29fe-4cdb-9dad-2197b9fa5837" (UID: "c25356ee-29fe-4cdb-9dad-2197b9fa5837"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:41.003911 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:41.003768    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25356ee-29fe-4cdb-9dad-2197b9fa5837-kube-api-access-46sz5" (OuterVolumeSpecName: "kube-api-access-46sz5") pod "c25356ee-29fe-4cdb-9dad-2197b9fa5837" (UID: "c25356ee-29fe-4cdb-9dad-2197b9fa5837"). InnerVolumeSpecName "kube-api-access-46sz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:41.108821 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:41.102260    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25356ee-29fe-4cdb-9dad-2197b9fa5837-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:19:41.108821 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:41.102342    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46sz5\" (UniqueName: \"kubernetes.io/projected/c25356ee-29fe-4cdb-9dad-2197b9fa5837-kube-api-access-46sz5\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:19:41.913854 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:41.913802    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654" event={"ID":"c25356ee-29fe-4cdb-9dad-2197b9fa5837","Type":"ContainerDied","Data":"77e0713db1992b574736382db92d146fb379bf9f2322cd00b0d8abaa961958b2"}
Sep 16 19:19:41.914492 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:41.913868    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"
Sep 16 19:19:41.914492 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:41.913888    2582 scope.go:117] "RemoveContainer" containerID="09175ac30f061ef4e9182bd5bb2fa429f3c1358bc1af8ec0ad6f4cd20c96a7e9"
Sep 16 19:19:41.937864 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:41.937819    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"]
Sep 16 19:19:41.941976 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:41.941945    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wx654"]
Sep 16 19:19:42.705244 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:42.705201    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c25356ee-29fe-4cdb-9dad-2197b9fa5837" path="/var/lib/kubelet/pods/c25356ee-29fe-4cdb-9dad-2197b9fa5837/volumes"
Sep 16 19:19:44.552735 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:44.552675    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:44.646009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:44.645969    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:45.969977 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:45.969932    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:48.704634 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.704601    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"]
Sep 16 19:19:48.705070 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.704964    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c25356ee-29fe-4cdb-9dad-2197b9fa5837" containerName="main"
Sep 16 19:19:48.705070 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.704977    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c25356ee-29fe-4cdb-9dad-2197b9fa5837" containerName="main"
Sep 16 19:19:48.705070 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.705039    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c25356ee-29fe-4cdb-9dad-2197b9fa5837" containerName="main"
Sep 16 19:19:48.708139 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.708113    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.711087 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.711058    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:19:48.712470 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.712446    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 19:19:48.712658 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.712593    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 19:19:48.712658 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.712567    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 19:19:48.712806 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.712567    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 19:19:48.712806 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.712731    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-cv956\""
Sep 16 19:19:48.717982 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.717954    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"]
Sep 16 19:19:48.777431 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.777386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34fd0d64-c172-4e54-ab7d-a5b9806b7532-tls-certs\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.777661 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.777439    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-home\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.777661 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.777568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-dshm\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.777661 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.777630    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.777800 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.777695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.777800 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.777719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-model-cache\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.777885 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.777816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcx2\" (UniqueName: \"kubernetes.io/projected/34fd0d64-c172-4e54-ab7d-a5b9806b7532-kube-api-access-tvcx2\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.777885 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.777843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34fd0d64-c172-4e54-ab7d-a5b9806b7532-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.794911 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.794875    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:48.878818 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.878767    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcx2\" (UniqueName: \"kubernetes.io/projected/34fd0d64-c172-4e54-ab7d-a5b9806b7532-kube-api-access-tvcx2\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.878818 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.878820    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34fd0d64-c172-4e54-ab7d-a5b9806b7532-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.879153 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.878846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34fd0d64-c172-4e54-ab7d-a5b9806b7532-tls-certs\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.879153 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.878868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-home\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.879153 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.878904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-dshm\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.879153 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.878921    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.879153 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.878967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.879153 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.878995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-model-cache\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.879510 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.879479    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-home\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.879708 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.879563    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-model-cache\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.879708 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.879500    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.879818 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.879709    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.879865 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.879825    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34fd0d64-c172-4e54-ab7d-a5b9806b7532-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.881942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.881920    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-dshm\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.882243 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.882224    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34fd0d64-c172-4e54-ab7d-a5b9806b7532-tls-certs\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.890169 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.890136    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcx2\" (UniqueName: \"kubernetes.io/projected/34fd0d64-c172-4e54-ab7d-a5b9806b7532-kube-api-access-tvcx2\") pod \"tokenizer-template-args-test-kserve-7d598c7594-6d7q8\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:48.994078 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.993961    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"]
Sep 16 19:19:48.998828 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:48.998799    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"
Sep 16 19:19:49.001976 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.001938    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-dd5hg\""
Sep 16 19:19:49.006348 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.006306    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:49.008552 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.008475    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"]
Sep 16 19:19:49.020692 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.020662    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:49.066457 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.066418    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:49.081342 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.081288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qkp2\" (UniqueName: \"kubernetes.io/projected/dc3821ea-8878-43b8-9764-b5f8ce266b0d-kube-api-access-4qkp2\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8\" (UID: \"dc3821ea-8878-43b8-9764-b5f8ce266b0d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"
Sep 16 19:19:49.081507 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.081458    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3821ea-8878-43b8-9764-b5f8ce266b0d-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8\" (UID: \"dc3821ea-8878-43b8-9764-b5f8ce266b0d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"
Sep 16 19:19:49.182701 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.182663    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3821ea-8878-43b8-9764-b5f8ce266b0d-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8\" (UID: \"dc3821ea-8878-43b8-9764-b5f8ce266b0d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"
Sep 16 19:19:49.182882 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.182771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qkp2\" (UniqueName: \"kubernetes.io/projected/dc3821ea-8878-43b8-9764-b5f8ce266b0d-kube-api-access-4qkp2\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8\" (UID: \"dc3821ea-8878-43b8-9764-b5f8ce266b0d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"
Sep 16 19:19:49.185565 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.185532    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3821ea-8878-43b8-9764-b5f8ce266b0d-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8\" (UID: \"dc3821ea-8878-43b8-9764-b5f8ce266b0d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"
Sep 16 19:19:49.192733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.192705    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qkp2\" (UniqueName: \"kubernetes.io/projected/dc3821ea-8878-43b8-9764-b5f8ce266b0d-kube-api-access-4qkp2\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8\" (UID: \"dc3821ea-8878-43b8-9764-b5f8ce266b0d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"
Sep 16 19:19:49.300389 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.300335    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"]
Sep 16 19:19:49.302817 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:19:49.302784    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34fd0d64_c172_4e54_ab7d_a5b9806b7532.slice/crio-20c32aa4c15e82fd873cf150bb168c449036bb03461956bc1c07569baa0001b3 WatchSource:0}: Error finding container 20c32aa4c15e82fd873cf150bb168c449036bb03461956bc1c07569baa0001b3: Status 404 returned error can't find the container with id 20c32aa4c15e82fd873cf150bb168c449036bb03461956bc1c07569baa0001b3
Sep 16 19:19:49.313092 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.313063    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"
Sep 16 19:19:49.596036 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.595999    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"]
Sep 16 19:19:49.599222 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:19:49.599189    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3821ea_8878_43b8_9764_b5f8ce266b0d.slice/crio-062d6e5104744991828227a2f2a2e0b564212ca3070c22b1ddd3be2dfd628d93 WatchSource:0}: Error finding container 062d6e5104744991828227a2f2a2e0b564212ca3070c22b1ddd3be2dfd628d93: Status 404 returned error can't find the container with id 062d6e5104744991828227a2f2a2e0b564212ca3070c22b1ddd3be2dfd628d93
Sep 16 19:19:49.961148 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.961020    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8" event={"ID":"dc3821ea-8878-43b8-9764-b5f8ce266b0d","Type":"ContainerStarted","Data":"8f6013a2241933ba0637dc4f1a6ac32bb13471171bdbc77b712b5949f8319959"}
Sep 16 19:19:49.961148 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.961089    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8" event={"ID":"dc3821ea-8878-43b8-9764-b5f8ce266b0d","Type":"ContainerStarted","Data":"062d6e5104744991828227a2f2a2e0b564212ca3070c22b1ddd3be2dfd628d93"}
Sep 16 19:19:49.961148 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.961133    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"
Sep 16 19:19:49.963686 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.963642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8" event={"ID":"34fd0d64-c172-4e54-ab7d-a5b9806b7532","Type":"ContainerStarted","Data":"5bb87ee77f743db12cdd56433b3ce79dbca76222e4c2b65124279bd22cf3e7f2"}
Sep 16 19:19:49.963880 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.963716    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8" event={"ID":"34fd0d64-c172-4e54-ab7d-a5b9806b7532","Type":"ContainerStarted","Data":"20c32aa4c15e82fd873cf150bb168c449036bb03461956bc1c07569baa0001b3"}
Sep 16 19:19:49.982449 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:49.982365    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8" podStartSLOduration=1.982338809 podStartE2EDuration="1.982338809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:49.981143887 +0000 UTC m=+2047.811752433" watchObservedRunningTime="2026-09-16 19:19:49.982338809 +0000 UTC m=+2047.812947354"
Sep 16 19:19:51.831913 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:51.831862    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:52.101424 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:52.101322    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:52.202062 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:52.202019    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:52.981122 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:52.981088    2582 generic.go:356] "Generic (PLEG): container finished" podID="34fd0d64-c172-4e54-ab7d-a5b9806b7532" containerID="5bb87ee77f743db12cdd56433b3ce79dbca76222e4c2b65124279bd22cf3e7f2" exitCode=0
Sep 16 19:19:52.981536 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:52.981174    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8" event={"ID":"34fd0d64-c172-4e54-ab7d-a5b9806b7532","Type":"ContainerDied","Data":"5bb87ee77f743db12cdd56433b3ce79dbca76222e4c2b65124279bd22cf3e7f2"}
Sep 16 19:19:53.988122 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:53.988084    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8" event={"ID":"34fd0d64-c172-4e54-ab7d-a5b9806b7532","Type":"ContainerStarted","Data":"78212ceb0d1724535d1e20a834ae634d1160d994f78b95efff124f42adbb3bea"}
Sep 16 19:19:54.008713 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:54.008658    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8" podStartSLOduration=3.4303547979999998 podStartE2EDuration="6.008639628s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.57828483s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:54.007370031 +0000 UTC m=+2051.837978576" watchObservedRunningTime="2026-09-16 19:19:54.008639628 +0000 UTC m=+2051.839248219"
Sep 16 19:19:54.021003 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:54.020962    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:57.427784 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:57.427739    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:57.527987 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:57.527947    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:59.021645 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:59.021595    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:59.034437 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:59.034402    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:19:59.959608 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:19:59.959523    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:00.028847 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:00.028814    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:20:01.582594 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:01.582546    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"]
Sep 16 19:20:01.608983 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:01.608945    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:16.602400 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.602362    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5" podUID="8aadd537-e93b-4bb6-ac12-38a078e1deb2" containerName="main" containerID="cri-o://ac74b9dedeaaaadb162feb04413256328d36f040ccfb9388c0ecdfa046a51c58" gracePeriod=45
Sep 16 19:20:16.849378 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.849354    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:20:16.917057 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.916999    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-tmp-dir\") pod \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") "
Sep 16 19:20:16.917057 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.917034    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-dshm\") pod \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") "
Sep 16 19:20:16.917057 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.917053    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aadd537-e93b-4bb6-ac12-38a078e1deb2-tls-certs\") pod \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") "
Sep 16 19:20:16.917241 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.917138    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmkqv\" (UniqueName: \"kubernetes.io/projected/8aadd537-e93b-4bb6-ac12-38a078e1deb2-kube-api-access-kmkqv\") pod \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") "
Sep 16 19:20:16.917241 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.917164    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-home\") pod \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") "
Sep 16 19:20:16.917241 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.917192    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-model-cache\") pod \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\" (UID: \"8aadd537-e93b-4bb6-ac12-38a078e1deb2\") "
Sep 16 19:20:16.917397 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.917271    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8aadd537-e93b-4bb6-ac12-38a078e1deb2" (UID: "8aadd537-e93b-4bb6-ac12-38a078e1deb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:16.917474 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.917442    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-home" (OuterVolumeSpecName: "home") pod "8aadd537-e93b-4bb6-ac12-38a078e1deb2" (UID: "8aadd537-e93b-4bb6-ac12-38a078e1deb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:16.917531 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.917488    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-model-cache" (OuterVolumeSpecName: "model-cache") pod "8aadd537-e93b-4bb6-ac12-38a078e1deb2" (UID: "8aadd537-e93b-4bb6-ac12-38a078e1deb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:16.917632 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.917615    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:20:16.919406 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.919387    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-dshm" (OuterVolumeSpecName: "dshm") pod "8aadd537-e93b-4bb6-ac12-38a078e1deb2" (UID: "8aadd537-e93b-4bb6-ac12-38a078e1deb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:16.919472 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.919439    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aadd537-e93b-4bb6-ac12-38a078e1deb2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8aadd537-e93b-4bb6-ac12-38a078e1deb2" (UID: "8aadd537-e93b-4bb6-ac12-38a078e1deb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:16.919545 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:16.919523    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aadd537-e93b-4bb6-ac12-38a078e1deb2-kube-api-access-kmkqv" (OuterVolumeSpecName: "kube-api-access-kmkqv") pod "8aadd537-e93b-4bb6-ac12-38a078e1deb2" (UID: "8aadd537-e93b-4bb6-ac12-38a078e1deb2"). InnerVolumeSpecName "kube-api-access-kmkqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:17.018170 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:17.018142    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:20:17.018271 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:17.018170    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aadd537-e93b-4bb6-ac12-38a078e1deb2-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:20:17.018271 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:17.018188    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmkqv\" (UniqueName: \"kubernetes.io/projected/8aadd537-e93b-4bb6-ac12-38a078e1deb2-kube-api-access-kmkqv\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:20:17.018271 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:17.018203    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:20:17.018271 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:17.018219    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aadd537-e93b-4bb6-ac12-38a078e1deb2-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:20:17.101443 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:17.101420    2582 generic.go:356] "Generic (PLEG): container finished" podID="8aadd537-e93b-4bb6-ac12-38a078e1deb2" containerID="ac74b9dedeaaaadb162feb04413256328d36f040ccfb9388c0ecdfa046a51c58" exitCode=0
Sep 16 19:20:17.101552 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:17.101464    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"
Sep 16 19:20:17.101552 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:17.101501    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5" event={"ID":"8aadd537-e93b-4bb6-ac12-38a078e1deb2","Type":"ContainerDied","Data":"ac74b9dedeaaaadb162feb04413256328d36f040ccfb9388c0ecdfa046a51c58"}
Sep 16 19:20:17.101552 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:17.101547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5" event={"ID":"8aadd537-e93b-4bb6-ac12-38a078e1deb2","Type":"ContainerDied","Data":"7ceee2da3c0f129f9c1b0dd246898e9844fad08eb1908f67d0c20abf22d7c41b"}
Sep 16 19:20:17.101695 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:17.101593    2582 scope.go:117] "RemoveContainer" containerID="ac74b9dedeaaaadb162feb04413256328d36f040ccfb9388c0ecdfa046a51c58"
Sep 16 19:20:17.110237 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:17.110216    2582 scope.go:117] "RemoveContainer" containerID="ac74b9dedeaaaadb162feb04413256328d36f040ccfb9388c0ecdfa046a51c58"
Sep 16 19:20:17.110509 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:20:17.110492    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac74b9dedeaaaadb162feb04413256328d36f040ccfb9388c0ecdfa046a51c58\": container with ID starting with ac74b9dedeaaaadb162feb04413256328d36f040ccfb9388c0ecdfa046a51c58 not found: ID does not exist" containerID="ac74b9dedeaaaadb162feb04413256328d36f040ccfb9388c0ecdfa046a51c58"
Sep 16 19:20:17.110560 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:17.110518    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac74b9dedeaaaadb162feb04413256328d36f040ccfb9388c0ecdfa046a51c58"} err="failed to get container status \"ac74b9dedeaaaadb162feb04413256328d36f040ccfb9388c0ecdfa046a51c58\": rpc error: code = NotFound desc = could not find container \"ac74b9dedeaaaadb162feb04413256328d36f040ccfb9388c0ecdfa046a51c58\": container with ID starting with ac74b9dedeaaaadb162feb04413256328d36f040ccfb9388c0ecdfa046a51c58 not found: ID does not exist"
Sep 16 19:20:17.124502 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:17.124482    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"]
Sep 16 19:20:17.131035 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:17.131016    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-46nq5"]
Sep 16 19:20:18.704428 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:18.704387    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aadd537-e93b-4bb6-ac12-38a078e1deb2" path="/var/lib/kubelet/pods/8aadd537-e93b-4bb6-ac12-38a078e1deb2/volumes"
Sep 16 19:20:20.970399 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:20.970367    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"
Sep 16 19:20:24.949837 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:24.949803    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:25.060146 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:25.060118    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:25.155836 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:25.155797    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:30.022244 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:30.022207    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:31.126775 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:20:31.126735    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:03.921258 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.921214    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"]
Sep 16 19:21:03.921802 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.921569    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aadd537-e93b-4bb6-ac12-38a078e1deb2" containerName="main"
Sep 16 19:21:03.921802 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.921596    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aadd537-e93b-4bb6-ac12-38a078e1deb2" containerName="main"
Sep 16 19:21:03.921802 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.921668    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aadd537-e93b-4bb6-ac12-38a078e1deb2" containerName="main"
Sep 16 19:21:03.924706 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.924683    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:03.929821 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.929793    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 19:21:03.929821 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.929806    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 16 19:21:03.930059 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.929828    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 19:21:03.930059 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.929913    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-gff9j\""
Sep 16 19:21:03.937144 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.937121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:03.937269 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.937154    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:03.937269 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.937175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-model-cache\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:03.937269 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.937197    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-home\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:03.937386 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.937298    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-dshm\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:03.937386 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.937333    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdwkw\" (UniqueName: \"kubernetes.io/projected/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-kube-api-access-bdwkw\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:03.942426 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:03.942400    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"]
Sep 16 19:21:04.012885 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.012851    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:04.038379 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.038337    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:04.038379 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.038383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:04.038630 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.038403    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-model-cache\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:04.038630 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.038525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-home\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:04.038716 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.038697    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-dshm\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:04.038777 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.038753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwkw\" (UniqueName: \"kubernetes.io/projected/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-kube-api-access-bdwkw\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:04.038831 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.038785    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-model-cache\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:04.038874 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.038849    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:04.038910 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.038870    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-home\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:04.041079 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.041060    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-dshm\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:04.041357 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.041340    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:04.054595 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.054538    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdwkw\" (UniqueName: \"kubernetes.io/projected/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-kube-api-access-bdwkw\") pod \"decom-v1-kserve-6cb797fb75-k4pzc\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:04.235673 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.235539    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:04.518826 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:04.518794    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"]
Sep 16 19:21:04.521517 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:21:04.521479    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ccf80d_0f47_4c5b_a160_c23c7802a2d7.slice/crio-c4baba8b0f536bcd13855059952826f329b02068d6bad18b7aaf38a9d177132a WatchSource:0}: Error finding container c4baba8b0f536bcd13855059952826f329b02068d6bad18b7aaf38a9d177132a: Status 404 returned error can't find the container with id c4baba8b0f536bcd13855059952826f329b02068d6bad18b7aaf38a9d177132a
Sep 16 19:21:05.328357 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:05.328301    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc" event={"ID":"85ccf80d-0f47-4c5b-a160-c23c7802a2d7","Type":"ContainerStarted","Data":"3b39499dba5f6f77c3d5f60fd94b010fdc9352b2725594d687850734e5960a01"}
Sep 16 19:21:05.328853 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:05.328410    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc" event={"ID":"85ccf80d-0f47-4c5b-a160-c23c7802a2d7","Type":"ContainerStarted","Data":"c4baba8b0f536bcd13855059952826f329b02068d6bad18b7aaf38a9d177132a"}
Sep 16 19:21:05.357140 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:05.357076    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc" podStartSLOduration=2.357055372 podStartE2EDuration="2.357055372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:05.354300906 +0000 UTC m=+2123.184909475" watchObservedRunningTime="2026-09-16 19:21:05.357055372 +0000 UTC m=+2123.187664194"
Sep 16 19:21:06.236593 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:06.236540    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:07.730194 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:07.730163    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:09.129694 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:09.129651    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:09.198780 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:09.198752    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:13.452624 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:13.452570    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:13.552204 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:13.552168    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:14.235791 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:14.235741    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:14.248974 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:14.248943    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:14.384073 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:14.384039    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:21:24.362424 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:24.362387    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:26.059072 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.059042    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"]
Sep 16 19:21:26.060936 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.060906    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:26.068418 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.068391    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"]
Sep 16 19:21:26.078839 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:21:26.078801    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:21:26.078839 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:21:26.078839 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:21:26.078839 ip-10-0-142-173 kubenswrapper[2582]:  > containerID="78212ceb0d1724535d1e20a834ae634d1160d994f78b95efff124f42adbb3bea" cmd=["/bin/sleep","15"]
Sep 16 19:21:26.079094 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.078861    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8" message="" err=<
Sep 16 19:21:26.079094 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:21:26.079094 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:21:26.079094 ip-10-0-142-173 kubenswrapper[2582]:  >
Sep 16 19:21:26.079094 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:21:26.078877    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:21:26.079094 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:21:26.079094 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 19:21:26.079094 ip-10-0-142-173 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8" podUID="34fd0d64-c172-4e54-ab7d-a5b9806b7532" containerName="main" containerID="cri-o://78212ceb0d1724535d1e20a834ae634d1160d994f78b95efff124f42adbb3bea"
Sep 16 19:21:26.079094 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.078957    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8" podUID="34fd0d64-c172-4e54-ab7d-a5b9806b7532" containerName="main" containerID="cri-o://78212ceb0d1724535d1e20a834ae634d1160d994f78b95efff124f42adbb3bea" gracePeriod=60
Sep 16 19:21:26.132781 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.132754    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:26.427048 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.427024    2582 generic.go:356] "Generic (PLEG): container finished" podID="34fd0d64-c172-4e54-ab7d-a5b9806b7532" containerID="78212ceb0d1724535d1e20a834ae634d1160d994f78b95efff124f42adbb3bea" exitCode=0
Sep 16 19:21:26.427163 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.427098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8" event={"ID":"34fd0d64-c172-4e54-ab7d-a5b9806b7532","Type":"ContainerDied","Data":"78212ceb0d1724535d1e20a834ae634d1160d994f78b95efff124f42adbb3bea"}
Sep 16 19:21:26.427163 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.427139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8" event={"ID":"34fd0d64-c172-4e54-ab7d-a5b9806b7532","Type":"ContainerDied","Data":"20c32aa4c15e82fd873cf150bb168c449036bb03461956bc1c07569baa0001b3"}
Sep 16 19:21:26.427163 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.427154    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20c32aa4c15e82fd873cf150bb168c449036bb03461956bc1c07569baa0001b3"
Sep 16 19:21:26.428862 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.428847    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:21:26.550419 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.550394    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-tmp-dir\") pod \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") "
Sep 16 19:21:26.550542 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.550449    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-home\") pod \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") "
Sep 16 19:21:26.550542 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.550478    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-kserve-provision-location\") pod \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") "
Sep 16 19:21:26.550542 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.550505    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-dshm\") pod \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") "
Sep 16 19:21:26.550542 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.550527    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34fd0d64-c172-4e54-ab7d-a5b9806b7532-cabundle-cert\") pod \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") "
Sep 16 19:21:26.550842 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.550587    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34fd0d64-c172-4e54-ab7d-a5b9806b7532-tls-certs\") pod \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") "
Sep 16 19:21:26.550842 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.550618    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-model-cache\") pod \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") "
Sep 16 19:21:26.550842 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.550681    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34fd0d64-c172-4e54-ab7d-a5b9806b7532" (UID: "34fd0d64-c172-4e54-ab7d-a5b9806b7532"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.550842 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.550711    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvcx2\" (UniqueName: \"kubernetes.io/projected/34fd0d64-c172-4e54-ab7d-a5b9806b7532-kube-api-access-tvcx2\") pod \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\" (UID: \"34fd0d64-c172-4e54-ab7d-a5b9806b7532\") "
Sep 16 19:21:26.551030 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.550872    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-home" (OuterVolumeSpecName: "home") pod "34fd0d64-c172-4e54-ab7d-a5b9806b7532" (UID: "34fd0d64-c172-4e54-ab7d-a5b9806b7532"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.551030 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.550893    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-model-cache" (OuterVolumeSpecName: "model-cache") pod "34fd0d64-c172-4e54-ab7d-a5b9806b7532" (UID: "34fd0d64-c172-4e54-ab7d-a5b9806b7532"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.551030 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.551011    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.551187 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.551033    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.551187 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.551046    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.551372 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.551317    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34fd0d64-c172-4e54-ab7d-a5b9806b7532-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34fd0d64-c172-4e54-ab7d-a5b9806b7532" (UID: "34fd0d64-c172-4e54-ab7d-a5b9806b7532"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:26.553034 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.553003    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34fd0d64-c172-4e54-ab7d-a5b9806b7532-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34fd0d64-c172-4e54-ab7d-a5b9806b7532" (UID: "34fd0d64-c172-4e54-ab7d-a5b9806b7532"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:26.553124 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.553074    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34fd0d64-c172-4e54-ab7d-a5b9806b7532-kube-api-access-tvcx2" (OuterVolumeSpecName: "kube-api-access-tvcx2") pod "34fd0d64-c172-4e54-ab7d-a5b9806b7532" (UID: "34fd0d64-c172-4e54-ab7d-a5b9806b7532"). InnerVolumeSpecName "kube-api-access-tvcx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:26.553232 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.553212    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-dshm" (OuterVolumeSpecName: "dshm") pod "34fd0d64-c172-4e54-ab7d-a5b9806b7532" (UID: "34fd0d64-c172-4e54-ab7d-a5b9806b7532"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.572061 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.572038    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34fd0d64-c172-4e54-ab7d-a5b9806b7532" (UID: "34fd0d64-c172-4e54-ab7d-a5b9806b7532"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.651874 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.651813    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.651874 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.651838    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34fd0d64-c172-4e54-ab7d-a5b9806b7532-cabundle-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.651874 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.651852    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34fd0d64-c172-4e54-ab7d-a5b9806b7532-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.651874 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.651863    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvcx2\" (UniqueName: \"kubernetes.io/projected/34fd0d64-c172-4e54-ab7d-a5b9806b7532-kube-api-access-tvcx2\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.652060 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:26.651877    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34fd0d64-c172-4e54-ab7d-a5b9806b7532-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:21:27.432351 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:27.432322    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"
Sep 16 19:21:27.480523 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:27.480496    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"]
Sep 16 19:21:27.485553 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:27.485527    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-6d7q8"]
Sep 16 19:21:28.703420 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:28.703387    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34fd0d64-c172-4e54-ab7d-a5b9806b7532" path="/var/lib/kubelet/pods/34fd0d64-c172-4e54-ab7d-a5b9806b7532/volumes"
Sep 16 19:21:30.970606 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:30.970546    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8" podUID="dc3821ea-8878-43b8-9764-b5f8ce266b0d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:40.971064 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:40.970978    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8" podUID="dc3821ea-8878-43b8-9764-b5f8ce266b0d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:41.080740 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:41.080679    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8" podUID="dc3821ea-8878-43b8-9764-b5f8ce266b0d" containerName="main" containerID="cri-o://8f6013a2241933ba0637dc4f1a6ac32bb13471171bdbc77b712b5949f8319959" gracePeriod=45
Sep 16 19:21:50.970548 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:50.970509    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8" podUID="dc3821ea-8878-43b8-9764-b5f8ce266b0d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:50.971016 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:50.970624    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"
Sep 16 19:21:56.902403 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:21:56.902357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:00.971068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:00.971024    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8" podUID="dc3821ea-8878-43b8-9764-b5f8ce266b0d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:22:10.970492 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:10.970450    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8" podUID="dc3821ea-8878-43b8-9764-b5f8ce266b0d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:22:11.343802 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.343770    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"
Sep 16 19:22:11.452902 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.452867    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qkp2\" (UniqueName: \"kubernetes.io/projected/dc3821ea-8878-43b8-9764-b5f8ce266b0d-kube-api-access-4qkp2\") pod \"dc3821ea-8878-43b8-9764-b5f8ce266b0d\" (UID: \"dc3821ea-8878-43b8-9764-b5f8ce266b0d\") "
Sep 16 19:22:11.453105 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.452915    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3821ea-8878-43b8-9764-b5f8ce266b0d-tls-certs\") pod \"dc3821ea-8878-43b8-9764-b5f8ce266b0d\" (UID: \"dc3821ea-8878-43b8-9764-b5f8ce266b0d\") "
Sep 16 19:22:11.455247 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.455209    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3821ea-8878-43b8-9764-b5f8ce266b0d-kube-api-access-4qkp2" (OuterVolumeSpecName: "kube-api-access-4qkp2") pod "dc3821ea-8878-43b8-9764-b5f8ce266b0d" (UID: "dc3821ea-8878-43b8-9764-b5f8ce266b0d"). InnerVolumeSpecName "kube-api-access-4qkp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:11.455368 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.455315    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc3821ea-8878-43b8-9764-b5f8ce266b0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc3821ea-8878-43b8-9764-b5f8ce266b0d" (UID: "dc3821ea-8878-43b8-9764-b5f8ce266b0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:11.554128 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.554089    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qkp2\" (UniqueName: \"kubernetes.io/projected/dc3821ea-8878-43b8-9764-b5f8ce266b0d-kube-api-access-4qkp2\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:22:11.554128 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.554124    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3821ea-8878-43b8-9764-b5f8ce266b0d-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:22:11.647315 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.647265    2582 generic.go:356] "Generic (PLEG): container finished" podID="dc3821ea-8878-43b8-9764-b5f8ce266b0d" containerID="8f6013a2241933ba0637dc4f1a6ac32bb13471171bdbc77b712b5949f8319959" exitCode=0
Sep 16 19:22:11.647471 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.647323    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"
Sep 16 19:22:11.647471 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.647358    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8" event={"ID":"dc3821ea-8878-43b8-9764-b5f8ce266b0d","Type":"ContainerDied","Data":"8f6013a2241933ba0637dc4f1a6ac32bb13471171bdbc77b712b5949f8319959"}
Sep 16 19:22:11.647471 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.647415    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8" event={"ID":"dc3821ea-8878-43b8-9764-b5f8ce266b0d","Type":"ContainerDied","Data":"062d6e5104744991828227a2f2a2e0b564212ca3070c22b1ddd3be2dfd628d93"}
Sep 16 19:22:11.647471 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.647437    2582 scope.go:117] "RemoveContainer" containerID="8f6013a2241933ba0637dc4f1a6ac32bb13471171bdbc77b712b5949f8319959"
Sep 16 19:22:11.656883 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.656863    2582 scope.go:117] "RemoveContainer" containerID="8f6013a2241933ba0637dc4f1a6ac32bb13471171bdbc77b712b5949f8319959"
Sep 16 19:22:11.657221 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:22:11.657201    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f6013a2241933ba0637dc4f1a6ac32bb13471171bdbc77b712b5949f8319959\": container with ID starting with 8f6013a2241933ba0637dc4f1a6ac32bb13471171bdbc77b712b5949f8319959 not found: ID does not exist" containerID="8f6013a2241933ba0637dc4f1a6ac32bb13471171bdbc77b712b5949f8319959"
Sep 16 19:22:11.657270 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.657235    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f6013a2241933ba0637dc4f1a6ac32bb13471171bdbc77b712b5949f8319959"} err="failed to get container status \"8f6013a2241933ba0637dc4f1a6ac32bb13471171bdbc77b712b5949f8319959\": rpc error: code = NotFound desc = could not find container \"8f6013a2241933ba0637dc4f1a6ac32bb13471171bdbc77b712b5949f8319959\": container with ID starting with 8f6013a2241933ba0637dc4f1a6ac32bb13471171bdbc77b712b5949f8319959 not found: ID does not exist"
Sep 16 19:22:11.673347 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.673312    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"]
Sep 16 19:22:11.677756 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:11.677726    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dhc5b8"]
Sep 16 19:22:12.704056 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:12.704024    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc3821ea-8878-43b8-9764-b5f8ce266b0d" path="/var/lib/kubelet/pods/dc3821ea-8878-43b8-9764-b5f8ce266b0d/volumes"
Sep 16 19:22:13.089898 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:13.089864    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:14.029657 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:14.029619    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"]
Sep 16 19:22:14.396557 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:14.396474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:18.435641 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:18.435604    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:19.726419 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:19.726382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:25.388144 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:25.388101    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:25.475117 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:25.475076    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:29.050238 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.050167    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc" podUID="85ccf80d-0f47-4c5b-a160-c23c7802a2d7" containerName="main" containerID="cri-o://3b39499dba5f6f77c3d5f60fd94b010fdc9352b2725594d687850734e5960a01" gracePeriod=45
Sep 16 19:22:29.313880 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.313796    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:22:29.421402 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.421354    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-tls-certs\") pod \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") "
Sep 16 19:22:29.421402 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.421411    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-home\") pod \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") "
Sep 16 19:22:29.421746 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.421476    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-model-cache\") pod \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") "
Sep 16 19:22:29.421746 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.421507    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdwkw\" (UniqueName: \"kubernetes.io/projected/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-kube-api-access-bdwkw\") pod \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") "
Sep 16 19:22:29.421746 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.421555    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-dshm\") pod \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") "
Sep 16 19:22:29.421746 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.421604    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-tmp-dir\") pod \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\" (UID: \"85ccf80d-0f47-4c5b-a160-c23c7802a2d7\") "
Sep 16 19:22:29.421964 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.421761    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "85ccf80d-0f47-4c5b-a160-c23c7802a2d7" (UID: "85ccf80d-0f47-4c5b-a160-c23c7802a2d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:29.421964 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.421780    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-home" (OuterVolumeSpecName: "home") pod "85ccf80d-0f47-4c5b-a160-c23c7802a2d7" (UID: "85ccf80d-0f47-4c5b-a160-c23c7802a2d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:29.422371 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.422002    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:22:29.422371 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.422018    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:22:29.422371 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.422014    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85ccf80d-0f47-4c5b-a160-c23c7802a2d7" (UID: "85ccf80d-0f47-4c5b-a160-c23c7802a2d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:29.424199 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.424165    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-dshm" (OuterVolumeSpecName: "dshm") pod "85ccf80d-0f47-4c5b-a160-c23c7802a2d7" (UID: "85ccf80d-0f47-4c5b-a160-c23c7802a2d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:29.424340 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.424228    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-kube-api-access-bdwkw" (OuterVolumeSpecName: "kube-api-access-bdwkw") pod "85ccf80d-0f47-4c5b-a160-c23c7802a2d7" (UID: "85ccf80d-0f47-4c5b-a160-c23c7802a2d7"). InnerVolumeSpecName "kube-api-access-bdwkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:29.424340 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.424228    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85ccf80d-0f47-4c5b-a160-c23c7802a2d7" (UID: "85ccf80d-0f47-4c5b-a160-c23c7802a2d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:29.522625 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.522552    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:22:29.522625 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.522618    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdwkw\" (UniqueName: \"kubernetes.io/projected/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-kube-api-access-bdwkw\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:22:29.522625 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.522628    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:22:29.522625 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.522638    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ccf80d-0f47-4c5b-a160-c23c7802a2d7-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:22:29.737895 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.737767    2582 generic.go:356] "Generic (PLEG): container finished" podID="85ccf80d-0f47-4c5b-a160-c23c7802a2d7" containerID="3b39499dba5f6f77c3d5f60fd94b010fdc9352b2725594d687850734e5960a01" exitCode=0
Sep 16 19:22:29.737895 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.737819    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"
Sep 16 19:22:29.737895 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.737827    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc" event={"ID":"85ccf80d-0f47-4c5b-a160-c23c7802a2d7","Type":"ContainerDied","Data":"3b39499dba5f6f77c3d5f60fd94b010fdc9352b2725594d687850734e5960a01"}
Sep 16 19:22:29.737895 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.737885    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc" event={"ID":"85ccf80d-0f47-4c5b-a160-c23c7802a2d7","Type":"ContainerDied","Data":"c4baba8b0f536bcd13855059952826f329b02068d6bad18b7aaf38a9d177132a"}
Sep 16 19:22:29.738256 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.737913    2582 scope.go:117] "RemoveContainer" containerID="3b39499dba5f6f77c3d5f60fd94b010fdc9352b2725594d687850734e5960a01"
Sep 16 19:22:29.748496 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.748477    2582 scope.go:117] "RemoveContainer" containerID="3b39499dba5f6f77c3d5f60fd94b010fdc9352b2725594d687850734e5960a01"
Sep 16 19:22:29.748904 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:22:29.748881    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b39499dba5f6f77c3d5f60fd94b010fdc9352b2725594d687850734e5960a01\": container with ID starting with 3b39499dba5f6f77c3d5f60fd94b010fdc9352b2725594d687850734e5960a01 not found: ID does not exist" containerID="3b39499dba5f6f77c3d5f60fd94b010fdc9352b2725594d687850734e5960a01"
Sep 16 19:22:29.748985 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.748917    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b39499dba5f6f77c3d5f60fd94b010fdc9352b2725594d687850734e5960a01"} err="failed to get container status \"3b39499dba5f6f77c3d5f60fd94b010fdc9352b2725594d687850734e5960a01\": rpc error: code = NotFound desc = could not find container \"3b39499dba5f6f77c3d5f60fd94b010fdc9352b2725594d687850734e5960a01\": container with ID starting with 3b39499dba5f6f77c3d5f60fd94b010fdc9352b2725594d687850734e5960a01 not found: ID does not exist"
Sep 16 19:22:29.762693 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.762641    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"]
Sep 16 19:22:29.767886 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:29.767849    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-k4pzc"]
Sep 16 19:22:30.707334 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:30.707266    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ccf80d-0f47-4c5b-a160-c23c7802a2d7" path="/var/lib/kubelet/pods/85ccf80d-0f47-4c5b-a160-c23c7802a2d7/volumes"
Sep 16 19:22:32.117515 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.117474    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"]
Sep 16 19:22:32.118084 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.118059    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34fd0d64-c172-4e54-ab7d-a5b9806b7532" containerName="main"
Sep 16 19:22:32.118084 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.118083    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="34fd0d64-c172-4e54-ab7d-a5b9806b7532" containerName="main"
Sep 16 19:22:32.118204 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.118103    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34fd0d64-c172-4e54-ab7d-a5b9806b7532" containerName="storage-initializer"
Sep 16 19:22:32.118204 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.118113    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="34fd0d64-c172-4e54-ab7d-a5b9806b7532" containerName="storage-initializer"
Sep 16 19:22:32.118204 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.118129    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ccf80d-0f47-4c5b-a160-c23c7802a2d7" containerName="main"
Sep 16 19:22:32.118204 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.118137    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ccf80d-0f47-4c5b-a160-c23c7802a2d7" containerName="main"
Sep 16 19:22:32.118204 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.118159    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc3821ea-8878-43b8-9764-b5f8ce266b0d" containerName="main"
Sep 16 19:22:32.118204 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.118167    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc3821ea-8878-43b8-9764-b5f8ce266b0d" containerName="main"
Sep 16 19:22:32.118492 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.118260    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ccf80d-0f47-4c5b-a160-c23c7802a2d7" containerName="main"
Sep 16 19:22:32.118492 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.118276    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc3821ea-8878-43b8-9764-b5f8ce266b0d" containerName="main"
Sep 16 19:22:32.118492 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.118286    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="34fd0d64-c172-4e54-ab7d-a5b9806b7532" containerName="main"
Sep 16 19:22:32.123065 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.123035    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.126395 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.126368    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 16 19:22:32.126395 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.126368    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 16 19:22:32.127712 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.127693    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-h6dcx\""
Sep 16 19:22:32.127842 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.127761    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 16 19:22:32.134153 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.134122    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"]
Sep 16 19:22:32.146822 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.146793    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-model-cache\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.146822 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.146826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-dshm\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.147011 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.146850    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-tls-certs\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.147011 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.146924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gszxr\" (UniqueName: \"kubernetes.io/projected/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-kube-api-access-gszxr\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.147011 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.146971    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.147011 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.146995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-home\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.160346 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.160312    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:32.247455 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.247411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.247455 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.247450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-home\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.247747 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.247590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-model-cache\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.247747 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.247636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-dshm\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.247747 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.247667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-tls-certs\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.247747 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.247727    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gszxr\" (UniqueName: \"kubernetes.io/projected/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-kube-api-access-gszxr\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.247960 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.247885    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-home\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.247960 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.247912    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-model-cache\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.248066 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.247979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.250369 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.250336    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-dshm\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.250547 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.250529    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-tls-certs\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.257253 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.257234    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gszxr\" (UniqueName: \"kubernetes.io/projected/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-kube-api-access-gszxr\") pod \"leave-v1-kserve-65684d4f5-bwslm\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.435043 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.434923    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:32.721671 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.721644    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"]
Sep 16 19:22:32.725123 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:22:32.725086    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47838ec_83c7_42bb_9036_f8d2abe1aa3f.slice/crio-84f9b303ab8158a6b3ea58d92ed85f8845ff01d1cce62b66e530374353dbae9c WatchSource:0}: Error finding container 84f9b303ab8158a6b3ea58d92ed85f8845ff01d1cce62b66e530374353dbae9c: Status 404 returned error can't find the container with id 84f9b303ab8158a6b3ea58d92ed85f8845ff01d1cce62b66e530374353dbae9c
Sep 16 19:22:32.757368 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:32.757324    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm" event={"ID":"a47838ec-83c7-42bb-9036-f8d2abe1aa3f","Type":"ContainerStarted","Data":"84f9b303ab8158a6b3ea58d92ed85f8845ff01d1cce62b66e530374353dbae9c"}
Sep 16 19:22:33.764552 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:33.764493    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm" event={"ID":"a47838ec-83c7-42bb-9036-f8d2abe1aa3f","Type":"ContainerStarted","Data":"a046ed1bd31ecf4fcf854b084bdbeb5204ef2f60ec8997133a2d1af9da1d3155"}
Sep 16 19:22:33.787205 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:33.787128    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm" podStartSLOduration=1.787105021 podStartE2EDuration="1.787105021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:33.785237375 +0000 UTC m=+2211.615845927" watchObservedRunningTime="2026-09-16 19:22:33.787105021 +0000 UTC m=+2211.617713623"
Sep 16 19:22:34.435447 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:34.435403    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:42.435475 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:42.435411    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:42.448770 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:42.448734    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:42.819462 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:42.819431    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:22:44.146527 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:44.146476    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:46.039701 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.039655    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:46.252075 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.252035    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"]
Sep 16 19:22:46.256998 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.256976    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"
Sep 16 19:22:46.265032 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.265008    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 19:22:46.265871 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.265836    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 19:22:46.266222 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.265834    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-cjrz4\""
Sep 16 19:22:46.266360 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.265846    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 19:22:46.266360 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.265999    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-7j92v\""
Sep 16 19:22:46.268589 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.268550    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"]
Sep 16 19:22:46.291835 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.291714    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:46.384074 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.384032    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x975g\" (UniqueName: \"kubernetes.io/projected/a9512437-02a6-4fd4-9c5b-3de417acc748-kube-api-access-x975g\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt\" (UID: \"a9512437-02a6-4fd4-9c5b-3de417acc748\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"
Sep 16 19:22:46.384309 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.384089    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9512437-02a6-4fd4-9c5b-3de417acc748-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt\" (UID: \"a9512437-02a6-4fd4-9c5b-3de417acc748\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"
Sep 16 19:22:46.484964 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.484908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x975g\" (UniqueName: \"kubernetes.io/projected/a9512437-02a6-4fd4-9c5b-3de417acc748-kube-api-access-x975g\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt\" (UID: \"a9512437-02a6-4fd4-9c5b-3de417acc748\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"
Sep 16 19:22:46.484964 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.484965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9512437-02a6-4fd4-9c5b-3de417acc748-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt\" (UID: \"a9512437-02a6-4fd4-9c5b-3de417acc748\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"
Sep 16 19:22:46.488091 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.488058    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9512437-02a6-4fd4-9c5b-3de417acc748-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt\" (UID: \"a9512437-02a6-4fd4-9c5b-3de417acc748\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"
Sep 16 19:22:46.494441 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.494418    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x975g\" (UniqueName: \"kubernetes.io/projected/a9512437-02a6-4fd4-9c5b-3de417acc748-kube-api-access-x975g\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt\" (UID: \"a9512437-02a6-4fd4-9c5b-3de417acc748\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"
Sep 16 19:22:46.569544 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.569420    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"
Sep 16 19:22:46.852708 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:46.852679    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"]
Sep 16 19:22:46.855198 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:22:46.855167    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9512437_02a6_4fd4_9c5b_3de417acc748.slice/crio-ab3bb922581f794f6aa418bc570ba848bc3fc4749e173fb1667d7eeaa132c36f WatchSource:0}: Error finding container ab3bb922581f794f6aa418bc570ba848bc3fc4749e173fb1667d7eeaa132c36f: Status 404 returned error can't find the container with id ab3bb922581f794f6aa418bc570ba848bc3fc4749e173fb1667d7eeaa132c36f
Sep 16 19:22:47.838833 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:47.838777    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt" event={"ID":"a9512437-02a6-4fd4-9c5b-3de417acc748","Type":"ContainerStarted","Data":"d862073a9d795ec5ceb407df72a408f1d31f2315f2e15aee32f1425d0736f037"}
Sep 16 19:22:47.839407 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:47.838853    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"
Sep 16 19:22:47.839407 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:47.838871    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt" event={"ID":"a9512437-02a6-4fd4-9c5b-3de417acc748","Type":"ContainerStarted","Data":"ab3bb922581f794f6aa418bc570ba848bc3fc4749e173fb1667d7eeaa132c36f"}
Sep 16 19:22:47.860779 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:47.860715    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt" podStartSLOduration=1.8606947520000001 podStartE2EDuration="1.860694752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:47.858687478 +0000 UTC m=+2225.689296025" watchObservedRunningTime="2026-09-16 19:22:47.860694752 +0000 UTC m=+2225.691303296"
Sep 16 19:22:55.755730 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:22:55.755674    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:01.322154 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.322102    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"]
Sep 16 19:23:01.325999 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.325977    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.330547 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.330518    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 16 19:23:01.344051 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.344016    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"]
Sep 16 19:23:01.384341 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.384254    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:01.440291 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.440245    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whdvr\" (UniqueName: \"kubernetes.io/projected/895bb466-be96-4ff2-a641-ac4f7e42b8a4-kube-api-access-whdvr\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.440499 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.440317    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-model-cache\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.440499 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.440337    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.440499 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.440365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895bb466-be96-4ff2-a641-ac4f7e42b8a4-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.440499 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.440382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-dshm\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.440691 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.440495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-home\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.541461 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.541405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-model-cache\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.541461 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.541454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.541808 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.541494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895bb466-be96-4ff2-a641-ac4f7e42b8a4-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.541808 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.541623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-dshm\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.541808 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.541714    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-home\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.541808 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.541795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whdvr\" (UniqueName: \"kubernetes.io/projected/895bb466-be96-4ff2-a641-ac4f7e42b8a4-kube-api-access-whdvr\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.542024 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.542007    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.542085 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.542034    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-model-cache\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.542153 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.542133    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-home\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.544428 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.544409    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-dshm\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.544639 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.544623    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895bb466-be96-4ff2-a641-ac4f7e42b8a4-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.551360 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.551336    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whdvr\" (UniqueName: \"kubernetes.io/projected/895bb466-be96-4ff2-a641-ac4f7e42b8a4-kube-api-access-whdvr\") pod \"leave-v2-kserve-556b4dddf9-gspzb\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.639108 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.638975    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:01.929856 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:01.929815    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"]
Sep 16 19:23:01.932197 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:23:01.932154    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895bb466_be96_4ff2_a641_ac4f7e42b8a4.slice/crio-27bfd56928f68efcf7354906a54a751e4459fa656a60053036617133358b9cb8 WatchSource:0}: Error finding container 27bfd56928f68efcf7354906a54a751e4459fa656a60053036617133358b9cb8: Status 404 returned error can't find the container with id 27bfd56928f68efcf7354906a54a751e4459fa656a60053036617133358b9cb8
Sep 16 19:23:02.355821 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:02.355788    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:02.488273 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:02.488233    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:02.914965 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:02.914911    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb" event={"ID":"895bb466-be96-4ff2-a641-ac4f7e42b8a4","Type":"ContainerStarted","Data":"e42121de7e72ad9f325b2e06ff674e59665d00489b587a205e13e13d04f8c1c9"}
Sep 16 19:23:02.915232 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:02.914997    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb" event={"ID":"895bb466-be96-4ff2-a641-ac4f7e42b8a4","Type":"ContainerStarted","Data":"27bfd56928f68efcf7354906a54a751e4459fa656a60053036617133358b9cb8"}
Sep 16 19:23:02.936867 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:02.936801    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb" podStartSLOduration=1.936780079 podStartE2EDuration="1.936780079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:02.935371216 +0000 UTC m=+2240.765979762" watchObservedRunningTime="2026-09-16 19:23:02.936780079 +0000 UTC m=+2240.767388619"
Sep 16 19:23:03.640087 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:03.640036    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:11.640012 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:11.639961    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:11.653285 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:11.653256    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:11.971697 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:11.971593    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:17.255828 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:17.255781    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:18.845664 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:18.845624    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"
Sep 16 19:23:37.417056 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:37.417000    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"]
Sep 16 19:23:37.418688 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:37.418653    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:37.486775 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:37.486727    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:38.846003 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:38.845949    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt" podUID="a9512437-02a6-4fd4-9c5b-3de417acc748" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:38.918769 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:38.918727    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:38.995845 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:38.995807    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:42.563862 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:42.563824    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:42.672516 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:42.672472    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:42.745530 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:42.745485    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"]
Sep 16 19:23:42.768995 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:42.768950    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:42.866871 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:42.866771    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:42.992629 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:42.992589    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"]
Sep 16 19:23:43.024945 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:43.024911    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:47.641072 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:47.641028    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:48.846117 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:48.846069    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt" podUID="a9512437-02a6-4fd4-9c5b-3de417acc748" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:52.440524 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:52.440461    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt" podUID="a9512437-02a6-4fd4-9c5b-3de417acc748" containerName="main" containerID="cri-o://d862073a9d795ec5ceb407df72a408f1d31f2315f2e15aee32f1425d0736f037" gracePeriod=45
Sep 16 19:23:52.456705 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:52.456667    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:57.766780 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:57.766703    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm" podUID="a47838ec-83c7-42bb-9036-f8d2abe1aa3f" containerName="main" containerID="cri-o://a046ed1bd31ecf4fcf854b084bdbeb5204ef2f60ec8997133a2d1af9da1d3155" gracePeriod=45
Sep 16 19:23:57.808585 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:57.808540    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm" podUID="a47838ec-83c7-42bb-9036-f8d2abe1aa3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 16 19:23:58.013258 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.013214    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb" podUID="895bb466-be96-4ff2-a641-ac4f7e42b8a4" containerName="main" containerID="cri-o://e42121de7e72ad9f325b2e06ff674e59665d00489b587a205e13e13d04f8c1c9" gracePeriod=45
Sep 16 19:23:58.024718 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.024690    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:23:58.070283 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.070236    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-dshm\") pod \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") "
Sep 16 19:23:58.070466 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.070306    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-tmp-dir\") pod \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") "
Sep 16 19:23:58.070466 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.070337    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-model-cache\") pod \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") "
Sep 16 19:23:58.070466 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.070384    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gszxr\" (UniqueName: \"kubernetes.io/projected/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-kube-api-access-gszxr\") pod \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") "
Sep 16 19:23:58.070466 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.070432    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-home\") pod \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") "
Sep 16 19:23:58.070466 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.070462    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-tls-certs\") pod \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\" (UID: \"a47838ec-83c7-42bb-9036-f8d2abe1aa3f\") "
Sep 16 19:23:58.070755 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.070626    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a47838ec-83c7-42bb-9036-f8d2abe1aa3f" (UID: "a47838ec-83c7-42bb-9036-f8d2abe1aa3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:58.070755 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.070641    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a47838ec-83c7-42bb-9036-f8d2abe1aa3f" (UID: "a47838ec-83c7-42bb-9036-f8d2abe1aa3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:58.070755 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.070683    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-home" (OuterVolumeSpecName: "home") pod "a47838ec-83c7-42bb-9036-f8d2abe1aa3f" (UID: "a47838ec-83c7-42bb-9036-f8d2abe1aa3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:58.070909 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.070860    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.070909 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.070880    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.070909 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.070894    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.072800 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.072766    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-dshm" (OuterVolumeSpecName: "dshm") pod "a47838ec-83c7-42bb-9036-f8d2abe1aa3f" (UID: "a47838ec-83c7-42bb-9036-f8d2abe1aa3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:58.072932 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.072801    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-kube-api-access-gszxr" (OuterVolumeSpecName: "kube-api-access-gszxr") pod "a47838ec-83c7-42bb-9036-f8d2abe1aa3f" (UID: "a47838ec-83c7-42bb-9036-f8d2abe1aa3f"). InnerVolumeSpecName "kube-api-access-gszxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:58.072932 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.072893    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a47838ec-83c7-42bb-9036-f8d2abe1aa3f" (UID: "a47838ec-83c7-42bb-9036-f8d2abe1aa3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:58.172138 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.172103    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.172138 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.172134    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gszxr\" (UniqueName: \"kubernetes.io/projected/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-kube-api-access-gszxr\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.172332 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.172152    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47838ec-83c7-42bb-9036-f8d2abe1aa3f-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.202922 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.202885    2582 generic.go:356] "Generic (PLEG): container finished" podID="a47838ec-83c7-42bb-9036-f8d2abe1aa3f" containerID="a046ed1bd31ecf4fcf854b084bdbeb5204ef2f60ec8997133a2d1af9da1d3155" exitCode=0
Sep 16 19:23:58.202922 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.202935    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"
Sep 16 19:23:58.203211 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.202973    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm" event={"ID":"a47838ec-83c7-42bb-9036-f8d2abe1aa3f","Type":"ContainerDied","Data":"a046ed1bd31ecf4fcf854b084bdbeb5204ef2f60ec8997133a2d1af9da1d3155"}
Sep 16 19:23:58.203211 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.203076    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm" event={"ID":"a47838ec-83c7-42bb-9036-f8d2abe1aa3f","Type":"ContainerDied","Data":"84f9b303ab8158a6b3ea58d92ed85f8845ff01d1cce62b66e530374353dbae9c"}
Sep 16 19:23:58.203211 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.203108    2582 scope.go:117] "RemoveContainer" containerID="a046ed1bd31ecf4fcf854b084bdbeb5204ef2f60ec8997133a2d1af9da1d3155"
Sep 16 19:23:58.205242 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.205210    2582 generic.go:356] "Generic (PLEG): container finished" podID="895bb466-be96-4ff2-a641-ac4f7e42b8a4" containerID="e42121de7e72ad9f325b2e06ff674e59665d00489b587a205e13e13d04f8c1c9" exitCode=0
Sep 16 19:23:58.205386 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.205286    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb" event={"ID":"895bb466-be96-4ff2-a641-ac4f7e42b8a4","Type":"ContainerDied","Data":"e42121de7e72ad9f325b2e06ff674e59665d00489b587a205e13e13d04f8c1c9"}
Sep 16 19:23:58.214087 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.214064    2582 scope.go:117] "RemoveContainer" containerID="a046ed1bd31ecf4fcf854b084bdbeb5204ef2f60ec8997133a2d1af9da1d3155"
Sep 16 19:23:58.214431 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:23:58.214412    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a046ed1bd31ecf4fcf854b084bdbeb5204ef2f60ec8997133a2d1af9da1d3155\": container with ID starting with a046ed1bd31ecf4fcf854b084bdbeb5204ef2f60ec8997133a2d1af9da1d3155 not found: ID does not exist" containerID="a046ed1bd31ecf4fcf854b084bdbeb5204ef2f60ec8997133a2d1af9da1d3155"
Sep 16 19:23:58.214499 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.214443    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a046ed1bd31ecf4fcf854b084bdbeb5204ef2f60ec8997133a2d1af9da1d3155"} err="failed to get container status \"a046ed1bd31ecf4fcf854b084bdbeb5204ef2f60ec8997133a2d1af9da1d3155\": rpc error: code = NotFound desc = could not find container \"a046ed1bd31ecf4fcf854b084bdbeb5204ef2f60ec8997133a2d1af9da1d3155\": container with ID starting with a046ed1bd31ecf4fcf854b084bdbeb5204ef2f60ec8997133a2d1af9da1d3155 not found: ID does not exist"
Sep 16 19:23:58.227913 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.227875    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"]
Sep 16 19:23:58.241211 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.241180    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bwslm"]
Sep 16 19:23:58.250591 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.250543    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:58.273239 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.273143    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-dshm\") pod \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") "
Sep 16 19:23:58.273239 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.273199    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-model-cache\") pod \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") "
Sep 16 19:23:58.273239 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.273222    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-home\") pod \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") "
Sep 16 19:23:58.273518 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.273277    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895bb466-be96-4ff2-a641-ac4f7e42b8a4-tls-certs\") pod \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") "
Sep 16 19:23:58.273518 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.273351    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whdvr\" (UniqueName: \"kubernetes.io/projected/895bb466-be96-4ff2-a641-ac4f7e42b8a4-kube-api-access-whdvr\") pod \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") "
Sep 16 19:23:58.273518 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.273393    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-tmp-dir\") pod \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\" (UID: \"895bb466-be96-4ff2-a641-ac4f7e42b8a4\") "
Sep 16 19:23:58.273518 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.273491    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "895bb466-be96-4ff2-a641-ac4f7e42b8a4" (UID: "895bb466-be96-4ff2-a641-ac4f7e42b8a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:58.273768 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.273675    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.273905 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.273880    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "895bb466-be96-4ff2-a641-ac4f7e42b8a4" (UID: "895bb466-be96-4ff2-a641-ac4f7e42b8a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:58.274038 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.274027    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-home" (OuterVolumeSpecName: "home") pod "895bb466-be96-4ff2-a641-ac4f7e42b8a4" (UID: "895bb466-be96-4ff2-a641-ac4f7e42b8a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:58.275947 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.275914    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/895bb466-be96-4ff2-a641-ac4f7e42b8a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "895bb466-be96-4ff2-a641-ac4f7e42b8a4" (UID: "895bb466-be96-4ff2-a641-ac4f7e42b8a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:58.276207 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.276189    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-dshm" (OuterVolumeSpecName: "dshm") pod "895bb466-be96-4ff2-a641-ac4f7e42b8a4" (UID: "895bb466-be96-4ff2-a641-ac4f7e42b8a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:58.276304 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.276234    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895bb466-be96-4ff2-a641-ac4f7e42b8a4-kube-api-access-whdvr" (OuterVolumeSpecName: "kube-api-access-whdvr") pod "895bb466-be96-4ff2-a641-ac4f7e42b8a4" (UID: "895bb466-be96-4ff2-a641-ac4f7e42b8a4"). InnerVolumeSpecName "kube-api-access-whdvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:58.374261 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.374219    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whdvr\" (UniqueName: \"kubernetes.io/projected/895bb466-be96-4ff2-a641-ac4f7e42b8a4-kube-api-access-whdvr\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.374261 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.374253    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.374261 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.374263    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.374261 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.374272    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895bb466-be96-4ff2-a641-ac4f7e42b8a4-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.374538 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.374281    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895bb466-be96-4ff2-a641-ac4f7e42b8a4-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:23:58.703836 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.703746    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a47838ec-83c7-42bb-9036-f8d2abe1aa3f" path="/var/lib/kubelet/pods/a47838ec-83c7-42bb-9036-f8d2abe1aa3f/volumes"
Sep 16 19:23:58.846081 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.846038    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt" podUID="a9512437-02a6-4fd4-9c5b-3de417acc748" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:58.846463 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:58.846164    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"
Sep 16 19:23:59.212300 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.212271    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"
Sep 16 19:23:59.212481 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.212270    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb" event={"ID":"895bb466-be96-4ff2-a641-ac4f7e42b8a4","Type":"ContainerDied","Data":"27bfd56928f68efcf7354906a54a751e4459fa656a60053036617133358b9cb8"}
Sep 16 19:23:59.212481 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.212409    2582 scope.go:117] "RemoveContainer" containerID="e42121de7e72ad9f325b2e06ff674e59665d00489b587a205e13e13d04f8c1c9"
Sep 16 19:23:59.230820 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.230787    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"]
Sep 16 19:23:59.236035 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.236004    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-gspzb"]
Sep 16 19:23:59.898654 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.898618    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"]
Sep 16 19:23:59.899026 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.898963    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a47838ec-83c7-42bb-9036-f8d2abe1aa3f" containerName="main"
Sep 16 19:23:59.899026 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.898975    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a47838ec-83c7-42bb-9036-f8d2abe1aa3f" containerName="main"
Sep 16 19:23:59.899026 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.898993    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="895bb466-be96-4ff2-a641-ac4f7e42b8a4" containerName="main"
Sep 16 19:23:59.899026 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.898998    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="895bb466-be96-4ff2-a641-ac4f7e42b8a4" containerName="main"
Sep 16 19:23:59.899160 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.899068    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a47838ec-83c7-42bb-9036-f8d2abe1aa3f" containerName="main"
Sep 16 19:23:59.899160 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.899079    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="895bb466-be96-4ff2-a641-ac4f7e42b8a4" containerName="main"
Sep 16 19:23:59.903507 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.903485    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:23:59.908071 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.908043    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 16 19:23:59.908071 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.908044    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 16 19:23:59.908262 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.908046    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-sgp6q\""
Sep 16 19:23:59.908262 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.908090    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 16 19:23:59.913600 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.913557    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"]
Sep 16 19:23:59.938743 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.938708    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:59.988206 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.988169    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54ab4e86-c68b-4fec-ba99-f8644088d920-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:23:59.988206 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.988221    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt2fc\" (UniqueName: \"kubernetes.io/projected/54ab4e86-c68b-4fec-ba99-f8644088d920-kube-api-access-gt2fc\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:23:59.988448 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.988259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:23:59.988448 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.988308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-dshm\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:23:59.988448 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.988388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-home\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:23:59.988448 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:23:59.988438    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-model-cache\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:00.089194 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.089148    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:00.089194 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.089190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-dshm\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:00.089483 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.089226    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-home\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:00.089483 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.089258    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-model-cache\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:00.089483 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.089312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54ab4e86-c68b-4fec-ba99-f8644088d920-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:00.089483 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.089356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt2fc\" (UniqueName: \"kubernetes.io/projected/54ab4e86-c68b-4fec-ba99-f8644088d920-kube-api-access-gt2fc\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:00.089731 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.089691    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-home\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:00.089731 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.089719    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:00.089859 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.089796    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-model-cache\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:00.091897 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.091872    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-dshm\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:00.092138 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.092121    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54ab4e86-c68b-4fec-ba99-f8644088d920-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:00.098529 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.098498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt2fc\" (UniqueName: \"kubernetes.io/projected/54ab4e86-c68b-4fec-ba99-f8644088d920-kube-api-access-gt2fc\") pod \"tri-v2-kserve-dcb554bf5-gg7sx\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:00.215635 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.215507    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:00.500065 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.499638    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"]
Sep 16 19:24:00.500065 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:24:00.499928    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ab4e86_c68b_4fec_ba99_f8644088d920.slice/crio-17e53121748d5e39b57896cf41b521e72e369fbc59d07e84efff04a74b1abec9 WatchSource:0}: Error finding container 17e53121748d5e39b57896cf41b521e72e369fbc59d07e84efff04a74b1abec9: Status 404 returned error can't find the container with id 17e53121748d5e39b57896cf41b521e72e369fbc59d07e84efff04a74b1abec9
Sep 16 19:24:00.707962 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:00.707848    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="895bb466-be96-4ff2-a641-ac4f7e42b8a4" path="/var/lib/kubelet/pods/895bb466-be96-4ff2-a641-ac4f7e42b8a4/volumes"
Sep 16 19:24:01.224320 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:01.224276    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx" event={"ID":"54ab4e86-c68b-4fec-ba99-f8644088d920","Type":"ContainerStarted","Data":"4797c9d78ec8e0f2c1b655e0ee7e0f3752367ad68e375662c7ec696baf0f16b0"}
Sep 16 19:24:01.224748 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:01.224338    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx" event={"ID":"54ab4e86-c68b-4fec-ba99-f8644088d920","Type":"ContainerStarted","Data":"17e53121748d5e39b57896cf41b521e72e369fbc59d07e84efff04a74b1abec9"}
Sep 16 19:24:01.246299 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:01.246246    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx" podStartSLOduration=2.246226879 podStartE2EDuration="2.246226879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:01.245449567 +0000 UTC m=+2299.076058111" watchObservedRunningTime="2026-09-16 19:24:01.246226879 +0000 UTC m=+2299.076835455"
Sep 16 19:24:02.216211 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:02.216168    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:08.845747 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:08.845701    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt" podUID="a9512437-02a6-4fd4-9c5b-3de417acc748" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:24:10.216632 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:10.216595    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:10.229133 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:10.229103    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:10.278737 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:10.278707    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:24:14.604511 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.604476    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"]
Sep 16 19:24:14.608284 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.608262    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.611379 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.611351    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 16 19:24:14.620169 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.620143    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"]
Sep 16 19:24:14.646904 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.646875    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:14.727429 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.727403    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8beb2f94-9bf0-4136-927f-7475898a1e92-tls-certs\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.727429 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.727434    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-home\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.727636 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.727477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx4vs\" (UniqueName: \"kubernetes.io/projected/8beb2f94-9bf0-4136-927f-7475898a1e92-kube-api-access-qx4vs\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.727636 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.727528    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-dshm\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.727636 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.727550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-model-cache\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.727636 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.727608    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.828298 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.828267    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx4vs\" (UniqueName: \"kubernetes.io/projected/8beb2f94-9bf0-4136-927f-7475898a1e92-kube-api-access-qx4vs\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.828443 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.828411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-dshm\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.828486 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.828451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-model-cache\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.828539 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.828495    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.828677 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.828657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8beb2f94-9bf0-4136-927f-7475898a1e92-tls-certs\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.828739 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.828696    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-home\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.828999 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.828971    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-model-cache\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.829103 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.829003    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.829103 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.829038    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-home\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.830949 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.830928    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-dshm\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.831234 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.831216    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8beb2f94-9bf0-4136-927f-7475898a1e92-tls-certs\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.837333 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.837316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx4vs\" (UniqueName: \"kubernetes.io/projected/8beb2f94-9bf0-4136-927f-7475898a1e92-kube-api-access-qx4vs\") pod \"tri-v3-kserve-798c88c66c-8s9q6\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:14.920050 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:14.919995    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:15.191398 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:15.191325    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"]
Sep 16 19:24:15.193730 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:24:15.193701    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8beb2f94_9bf0_4136_927f_7475898a1e92.slice/crio-b9c54a7da695eafccb5ce66d6a997f3e045b72ff3124fa533441b012da1cec0c WatchSource:0}: Error finding container b9c54a7da695eafccb5ce66d6a997f3e045b72ff3124fa533441b012da1cec0c: Status 404 returned error can't find the container with id b9c54a7da695eafccb5ce66d6a997f3e045b72ff3124fa533441b012da1cec0c
Sep 16 19:24:15.294760 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:15.294724    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6" event={"ID":"8beb2f94-9bf0-4136-927f-7475898a1e92","Type":"ContainerStarted","Data":"040c8fa5b0f547629d1fa67e9e45682727c4b630044989acc9f7bd54dd98b5d1"}
Sep 16 19:24:15.294904 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:15.294779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6" event={"ID":"8beb2f94-9bf0-4136-927f-7475898a1e92","Type":"ContainerStarted","Data":"b9c54a7da695eafccb5ce66d6a997f3e045b72ff3124fa533441b012da1cec0c"}
Sep 16 19:24:15.316759 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:15.316696    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6" podStartSLOduration=1.316673981 podStartE2EDuration="1.316673981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:15.315153495 +0000 UTC m=+2313.145762041" watchObservedRunningTime="2026-09-16 19:24:15.316673981 +0000 UTC m=+2313.147282527"
Sep 16 19:24:15.920523 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:15.920483    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:18.845707 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:18.845659    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt" podUID="a9512437-02a6-4fd4-9c5b-3de417acc748" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:24:22.546728 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:22.546694    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:22.683460 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:22.683432    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"
Sep 16 19:24:22.795828 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:22.795788    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x975g\" (UniqueName: \"kubernetes.io/projected/a9512437-02a6-4fd4-9c5b-3de417acc748-kube-api-access-x975g\") pod \"a9512437-02a6-4fd4-9c5b-3de417acc748\" (UID: \"a9512437-02a6-4fd4-9c5b-3de417acc748\") "
Sep 16 19:24:22.796022 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:22.795850    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9512437-02a6-4fd4-9c5b-3de417acc748-tls-certs\") pod \"a9512437-02a6-4fd4-9c5b-3de417acc748\" (UID: \"a9512437-02a6-4fd4-9c5b-3de417acc748\") "
Sep 16 19:24:22.798323 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:22.798293    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9512437-02a6-4fd4-9c5b-3de417acc748-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9512437-02a6-4fd4-9c5b-3de417acc748" (UID: "a9512437-02a6-4fd4-9c5b-3de417acc748"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:22.798429 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:22.798346    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9512437-02a6-4fd4-9c5b-3de417acc748-kube-api-access-x975g" (OuterVolumeSpecName: "kube-api-access-x975g") pod "a9512437-02a6-4fd4-9c5b-3de417acc748" (UID: "a9512437-02a6-4fd4-9c5b-3de417acc748"). InnerVolumeSpecName "kube-api-access-x975g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:22.897705 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:22.897669    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9512437-02a6-4fd4-9c5b-3de417acc748-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:24:22.897705 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:22.897697    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x975g\" (UniqueName: \"kubernetes.io/projected/a9512437-02a6-4fd4-9c5b-3de417acc748-kube-api-access-x975g\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:24:23.335629 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:23.335592    2582 generic.go:356] "Generic (PLEG): container finished" podID="a9512437-02a6-4fd4-9c5b-3de417acc748" containerID="d862073a9d795ec5ceb407df72a408f1d31f2315f2e15aee32f1425d0736f037" exitCode=0
Sep 16 19:24:23.335629 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:23.335629    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"
Sep 16 19:24:23.335920 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:23.335643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt" event={"ID":"a9512437-02a6-4fd4-9c5b-3de417acc748","Type":"ContainerDied","Data":"d862073a9d795ec5ceb407df72a408f1d31f2315f2e15aee32f1425d0736f037"}
Sep 16 19:24:23.335920 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:23.335700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt" event={"ID":"a9512437-02a6-4fd4-9c5b-3de417acc748","Type":"ContainerDied","Data":"ab3bb922581f794f6aa418bc570ba848bc3fc4749e173fb1667d7eeaa132c36f"}
Sep 16 19:24:23.335920 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:23.335731    2582 scope.go:117] "RemoveContainer" containerID="d862073a9d795ec5ceb407df72a408f1d31f2315f2e15aee32f1425d0736f037"
Sep 16 19:24:23.345261 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:23.345241    2582 scope.go:117] "RemoveContainer" containerID="d862073a9d795ec5ceb407df72a408f1d31f2315f2e15aee32f1425d0736f037"
Sep 16 19:24:23.345542 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:24:23.345523    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d862073a9d795ec5ceb407df72a408f1d31f2315f2e15aee32f1425d0736f037\": container with ID starting with d862073a9d795ec5ceb407df72a408f1d31f2315f2e15aee32f1425d0736f037 not found: ID does not exist" containerID="d862073a9d795ec5ceb407df72a408f1d31f2315f2e15aee32f1425d0736f037"
Sep 16 19:24:23.345620 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:23.345554    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d862073a9d795ec5ceb407df72a408f1d31f2315f2e15aee32f1425d0736f037"} err="failed to get container status \"d862073a9d795ec5ceb407df72a408f1d31f2315f2e15aee32f1425d0736f037\": rpc error: code = NotFound desc = could not find container \"d862073a9d795ec5ceb407df72a408f1d31f2315f2e15aee32f1425d0736f037\": container with ID starting with d862073a9d795ec5ceb407df72a408f1d31f2315f2e15aee32f1425d0736f037 not found: ID does not exist"
Sep 16 19:24:23.359142 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:23.359116    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"]
Sep 16 19:24:23.362134 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:23.362110    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schek8zmt"]
Sep 16 19:24:24.389693 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:24.389658    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:24.704998 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:24.704911    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9512437-02a6-4fd4-9c5b-3de417acc748" path="/var/lib/kubelet/pods/a9512437-02a6-4fd4-9c5b-3de417acc748/volumes"
Sep 16 19:24:24.920918 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:24.920873    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:24.934193 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:24.934150    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:25.358913 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:25.358875    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:24:25.891152 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:25.891112    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:25.990302 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:25.990266    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:30.034248 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:30.034214    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:30.100323 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:30.100294    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:31.825420 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:31.825384    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:37.559785 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:37.559700    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:37.659838 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:37.659801    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:49.523361 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.523307    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:49.740111 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.740078    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"]
Sep 16 19:24:49.740443 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.740430    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9512437-02a6-4fd4-9c5b-3de417acc748" containerName="main"
Sep 16 19:24:49.740443 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.740443    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9512437-02a6-4fd4-9c5b-3de417acc748" containerName="main"
Sep 16 19:24:49.740598 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.740511    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9512437-02a6-4fd4-9c5b-3de417acc748" containerName="main"
Sep 16 19:24:49.743545 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.743522    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"
Sep 16 19:24:49.747879 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.747852    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 19:24:49.748036 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.747883    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:24:49.748036 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.747904    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:24:49.748165 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.748063    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-8l9cd\""
Sep 16 19:24:49.748165 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.747952    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-x4kjb\""
Sep 16 19:24:49.754664 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.754635    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"]
Sep 16 19:24:49.768506 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.768479    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:49.839160 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.839044    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq79q\" (UniqueName: \"kubernetes.io/projected/ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c-kube-api-access-vq79q\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql\" (UID: \"ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"
Sep 16 19:24:49.839160 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.839097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql\" (UID: \"ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"
Sep 16 19:24:49.939885 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.939836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq79q\" (UniqueName: \"kubernetes.io/projected/ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c-kube-api-access-vq79q\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql\" (UID: \"ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"
Sep 16 19:24:49.939885 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.939889    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql\" (UID: \"ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"
Sep 16 19:24:49.942724 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.942701    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql\" (UID: \"ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"
Sep 16 19:24:49.949181 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:49.949156    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq79q\" (UniqueName: \"kubernetes.io/projected/ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c-kube-api-access-vq79q\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql\" (UID: \"ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"
Sep 16 19:24:50.056021 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:50.055975    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"
Sep 16 19:24:50.331691 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:50.331666    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"]
Sep 16 19:24:50.333973 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:24:50.333945    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddaf99db_2b0c_4b0f_abc3_2e22824d3a6c.slice/crio-58eef4999f120c14237714638d4aa1b88a5db611709df7f424046622c6d515db WatchSource:0}: Error finding container 58eef4999f120c14237714638d4aa1b88a5db611709df7f424046622c6d515db: Status 404 returned error can't find the container with id 58eef4999f120c14237714638d4aa1b88a5db611709df7f424046622c6d515db
Sep 16 19:24:50.468286 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:50.468238    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql" event={"ID":"ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c","Type":"ContainerStarted","Data":"82b16eb4999a5b21c509bf58a7b9c8da7948c776f47a55e3a6621aa63d302e93"}
Sep 16 19:24:50.468490 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:50.468309    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql" event={"ID":"ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c","Type":"ContainerStarted","Data":"58eef4999f120c14237714638d4aa1b88a5db611709df7f424046622c6d515db"}
Sep 16 19:24:50.468490 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:50.468371    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"
Sep 16 19:24:50.492215 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:24:50.492140    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql" podStartSLOduration=1.492117006 podStartE2EDuration="1.492117006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:50.489833389 +0000 UTC m=+2348.320441934" watchObservedRunningTime="2026-09-16 19:24:50.492117006 +0000 UTC m=+2348.322725554"
Sep 16 19:25:05.813165 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:05.813129    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:05.873291 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:05.873257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:21.475914 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:21.475887    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"
Sep 16 19:25:22.509199 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:22.509165    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:39.856740 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:39.856694    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:40.120439 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:40.120350    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"]
Sep 16 19:25:40.124616 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:40.124565    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"
Sep 16 19:25:40.129116 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:40.129082    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 19:25:40.129279 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:40.129086    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-sbkww\""
Sep 16 19:25:40.135012 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:40.134981    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"]
Sep 16 19:25:40.160379 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:40.160347    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:40.175309 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:40.175264    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa780fc9-79a9-4cde-8e35-b590efac9e68-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m\" (UID: \"aa780fc9-79a9-4cde-8e35-b590efac9e68\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"
Sep 16 19:25:40.175428 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:40.175374    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tncvz\" (UniqueName: \"kubernetes.io/projected/aa780fc9-79a9-4cde-8e35-b590efac9e68-kube-api-access-tncvz\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m\" (UID: \"aa780fc9-79a9-4cde-8e35-b590efac9e68\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"
Sep 16 19:25:40.276218 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:40.276180    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tncvz\" (UniqueName: \"kubernetes.io/projected/aa780fc9-79a9-4cde-8e35-b590efac9e68-kube-api-access-tncvz\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m\" (UID: \"aa780fc9-79a9-4cde-8e35-b590efac9e68\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"
Sep 16 19:25:40.276427 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:40.276274    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa780fc9-79a9-4cde-8e35-b590efac9e68-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m\" (UID: \"aa780fc9-79a9-4cde-8e35-b590efac9e68\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"
Sep 16 19:25:40.279209 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:40.279180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa780fc9-79a9-4cde-8e35-b590efac9e68-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m\" (UID: \"aa780fc9-79a9-4cde-8e35-b590efac9e68\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"
Sep 16 19:25:40.285392 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:40.285360    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tncvz\" (UniqueName: \"kubernetes.io/projected/aa780fc9-79a9-4cde-8e35-b590efac9e68-kube-api-access-tncvz\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m\" (UID: \"aa780fc9-79a9-4cde-8e35-b590efac9e68\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"
Sep 16 19:25:40.437550 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:40.437439    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"
Sep 16 19:25:40.713892 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:40.713866    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"]
Sep 16 19:25:40.716218 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:25:40.716191    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa780fc9_79a9_4cde_8e35_b590efac9e68.slice/crio-7ceb771fb833709c84d2d53b9d44e2d42f5f6c1e58d946e84e8c414c3e3a0555 WatchSource:0}: Error finding container 7ceb771fb833709c84d2d53b9d44e2d42f5f6c1e58d946e84e8c414c3e3a0555: Status 404 returned error can't find the container with id 7ceb771fb833709c84d2d53b9d44e2d42f5f6c1e58d946e84e8c414c3e3a0555
Sep 16 19:25:41.718870 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:41.718812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m" event={"ID":"aa780fc9-79a9-4cde-8e35-b590efac9e68","Type":"ContainerStarted","Data":"e809dc65c2b58e9259283d838fced1d3552e78022d4ed3c3b61e70662ea2a82d"}
Sep 16 19:25:41.719426 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:41.718919    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m" event={"ID":"aa780fc9-79a9-4cde-8e35-b590efac9e68","Type":"ContainerStarted","Data":"7ceb771fb833709c84d2d53b9d44e2d42f5f6c1e58d946e84e8c414c3e3a0555"}
Sep 16 19:25:41.719426 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:41.718951    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"
Sep 16 19:25:41.741154 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:41.741072    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m" podStartSLOduration=1.741045148 podStartE2EDuration="1.741045148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:41.739554784 +0000 UTC m=+2399.570163331" watchObservedRunningTime="2026-09-16 19:25:41.741045148 +0000 UTC m=+2399.571653695"
Sep 16 19:25:56.786812 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:25:56.786766    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:12.725259 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:26:12.725181    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"
Sep 16 19:26:22.930453 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:26:22.930407    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"]
Sep 16 19:26:22.931018 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:26:22.930744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:22.998762 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:26:22.998720    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:32.726090 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:26:32.726044    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m" podUID="aa780fc9-79a9-4cde-8e35-b590efac9e68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:37.953232 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:26:37.953160    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m" podUID="aa780fc9-79a9-4cde-8e35-b590efac9e68" containerName="main" containerID="cri-o://e809dc65c2b58e9259283d838fced1d3552e78022d4ed3c3b61e70662ea2a82d" gracePeriod=45
Sep 16 19:26:42.726062 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:26:42.726013    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m" podUID="aa780fc9-79a9-4cde-8e35-b590efac9e68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:43.585033 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:26:43.584998    2582 scope.go:117] "RemoveContainer" containerID="78212ceb0d1724535d1e20a834ae634d1160d994f78b95efff124f42adbb3bea"
Sep 16 19:26:43.597374 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:26:43.597341    2582 scope.go:117] "RemoveContainer" containerID="5bb87ee77f743db12cdd56433b3ce79dbca76222e4c2b65124279bd22cf3e7f2"
Sep 16 19:26:52.725499 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:26:52.725444    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m" podUID="aa780fc9-79a9-4cde-8e35-b590efac9e68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:52.726095 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:26:52.725654    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"
Sep 16 19:27:00.787392 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:00.787341    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:00.890889 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:00.890844    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:00.988980 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:00.988947    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:01.205949 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:01.205861    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:02.197514 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:02.197460    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"]
Sep 16 19:27:02.223045 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:02.222975    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:02.725759 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:02.725712    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m" podUID="aa780fc9-79a9-4cde-8e35-b590efac9e68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:27:02.799679 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:02.799633    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"]
Sep 16 19:27:02.823094 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:02.823058    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:05.850938 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:05.850897    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:08.157494 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:08.157445    2582 generic.go:356] "Generic (PLEG): container finished" podID="aa780fc9-79a9-4cde-8e35-b590efac9e68" containerID="e809dc65c2b58e9259283d838fced1d3552e78022d4ed3c3b61e70662ea2a82d" exitCode=0
Sep 16 19:27:08.157959 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:08.157540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m" event={"ID":"aa780fc9-79a9-4cde-8e35-b590efac9e68","Type":"ContainerDied","Data":"e809dc65c2b58e9259283d838fced1d3552e78022d4ed3c3b61e70662ea2a82d"}
Sep 16 19:27:08.208938 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:08.208905    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"
Sep 16 19:27:08.302554 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:08.302515    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa780fc9-79a9-4cde-8e35-b590efac9e68-tls-certs\") pod \"aa780fc9-79a9-4cde-8e35-b590efac9e68\" (UID: \"aa780fc9-79a9-4cde-8e35-b590efac9e68\") "
Sep 16 19:27:08.302840 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:08.302603    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tncvz\" (UniqueName: \"kubernetes.io/projected/aa780fc9-79a9-4cde-8e35-b590efac9e68-kube-api-access-tncvz\") pod \"aa780fc9-79a9-4cde-8e35-b590efac9e68\" (UID: \"aa780fc9-79a9-4cde-8e35-b590efac9e68\") "
Sep 16 19:27:08.305341 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:08.305305    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa780fc9-79a9-4cde-8e35-b590efac9e68-kube-api-access-tncvz" (OuterVolumeSpecName: "kube-api-access-tncvz") pod "aa780fc9-79a9-4cde-8e35-b590efac9e68" (UID: "aa780fc9-79a9-4cde-8e35-b590efac9e68"). InnerVolumeSpecName "kube-api-access-tncvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:08.305474 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:08.305306    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa780fc9-79a9-4cde-8e35-b590efac9e68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa780fc9-79a9-4cde-8e35-b590efac9e68" (UID: "aa780fc9-79a9-4cde-8e35-b590efac9e68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:08.404399 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:08.404364    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa780fc9-79a9-4cde-8e35-b590efac9e68-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:08.404399 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:08.404396    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tncvz\" (UniqueName: \"kubernetes.io/projected/aa780fc9-79a9-4cde-8e35-b590efac9e68-kube-api-access-tncvz\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:09.164855 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:09.164802    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m" event={"ID":"aa780fc9-79a9-4cde-8e35-b590efac9e68","Type":"ContainerDied","Data":"7ceb771fb833709c84d2d53b9d44e2d42f5f6c1e58d946e84e8c414c3e3a0555"}
Sep 16 19:27:09.164855 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:09.164841    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"
Sep 16 19:27:09.165599 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:09.164886    2582 scope.go:117] "RemoveContainer" containerID="e809dc65c2b58e9259283d838fced1d3552e78022d4ed3c3b61e70662ea2a82d"
Sep 16 19:27:09.185445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:09.185400    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"]
Sep 16 19:27:09.187427 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:09.187394    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schesgw5m"]
Sep 16 19:27:10.704096 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:10.704053    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa780fc9-79a9-4cde-8e35-b590efac9e68" path="/var/lib/kubelet/pods/aa780fc9-79a9-4cde-8e35-b590efac9e68/volumes"
Sep 16 19:27:12.115178 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.115140    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"]
Sep 16 19:27:12.115795 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.115774    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa780fc9-79a9-4cde-8e35-b590efac9e68" containerName="main"
Sep 16 19:27:12.115910 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.115799    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa780fc9-79a9-4cde-8e35-b590efac9e68" containerName="main"
Sep 16 19:27:12.115966 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.115924    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa780fc9-79a9-4cde-8e35-b590efac9e68" containerName="main"
Sep 16 19:27:12.120636 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.120611    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.123763 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.123740    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 16 19:27:12.125457 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.125263    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-5h9hc\""
Sep 16 19:27:12.125457 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.125275    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 16 19:27:12.125457 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.125346    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 16 19:27:12.128736 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.128712    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"]
Sep 16 19:27:12.135982 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.135954    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:12.142554 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.142513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.142554 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.142559    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-tls-certs\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.142779 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.142672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdtdj\" (UniqueName: \"kubernetes.io/projected/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-kube-api-access-qdtdj\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.142779 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.142732    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-model-cache\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.142779 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.142767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-home\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.142909 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.142798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-dshm\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.243298 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.243248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtdj\" (UniqueName: \"kubernetes.io/projected/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-kube-api-access-qdtdj\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.243610 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.243332    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-model-cache\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.243610 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.243364    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-home\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.243610 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.243404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-dshm\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.243610 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.243459    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.243610 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.243483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-tls-certs\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.243921 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.243874    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-model-cache\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.243980 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.243921    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-home\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.243980 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.243944    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.246376 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.246344    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-dshm\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.246709 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.246691    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-tls-certs\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.252758 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.252736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdtdj\" (UniqueName: \"kubernetes.io/projected/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-kube-api-access-qdtdj\") pod \"late-v1-kserve-56cbc8456c-2655g\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.433057 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.432921    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:12.716001 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:12.715895    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"]
Sep 16 19:27:12.720323 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:27:12.720292    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1fb9c4_6cdb_4dde_b911_49d4b467e40b.slice/crio-9afdc5b0155654b94f51ba46c15c0f1f1df3bab3871c440194da10dde2c84ac6 WatchSource:0}: Error finding container 9afdc5b0155654b94f51ba46c15c0f1f1df3bab3871c440194da10dde2c84ac6: Status 404 returned error can't find the container with id 9afdc5b0155654b94f51ba46c15c0f1f1df3bab3871c440194da10dde2c84ac6
Sep 16 19:27:13.187969 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:13.187922    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g" event={"ID":"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b","Type":"ContainerStarted","Data":"542f749d212021577d06d2ff5161f74a5c908c573820d6e1d1cec90f4fcead94"}
Sep 16 19:27:13.188511 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:13.187991    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g" event={"ID":"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b","Type":"ContainerStarted","Data":"9afdc5b0155654b94f51ba46c15c0f1f1df3bab3871c440194da10dde2c84ac6"}
Sep 16 19:27:13.209631 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:13.209534    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g" podStartSLOduration=1.209508122 podStartE2EDuration="1.209508122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:13.208386034 +0000 UTC m=+2491.038994616" watchObservedRunningTime="2026-09-16 19:27:13.209508122 +0000 UTC m=+2491.040116668"
Sep 16 19:27:13.433643 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:13.433563    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:13.853630 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:13.853595    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:13.864883 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:13.864839    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"]
Sep 16 19:27:13.928556 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:13.928523    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:17.226138 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.226079    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx" podUID="54ab4e86-c68b-4fec-ba99-f8644088d920" containerName="main" containerID="cri-o://4797c9d78ec8e0f2c1b655e0ee7e0f3752367ad68e375662c7ec696baf0f16b0" gracePeriod=45
Sep 16 19:27:17.268445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.268391    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx" podUID="54ab4e86-c68b-4fec-ba99-f8644088d920" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 19:27:17.504641 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.504605    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:27:17.607476 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.607431    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-dshm\") pod \"54ab4e86-c68b-4fec-ba99-f8644088d920\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") "
Sep 16 19:27:17.607476 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.607489    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-tmp-dir\") pod \"54ab4e86-c68b-4fec-ba99-f8644088d920\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") "
Sep 16 19:27:17.607842 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.607518    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt2fc\" (UniqueName: \"kubernetes.io/projected/54ab4e86-c68b-4fec-ba99-f8644088d920-kube-api-access-gt2fc\") pod \"54ab4e86-c68b-4fec-ba99-f8644088d920\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") "
Sep 16 19:27:17.607842 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.607543    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-model-cache\") pod \"54ab4e86-c68b-4fec-ba99-f8644088d920\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") "
Sep 16 19:27:17.607842 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.607569    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54ab4e86-c68b-4fec-ba99-f8644088d920-tls-certs\") pod \"54ab4e86-c68b-4fec-ba99-f8644088d920\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") "
Sep 16 19:27:17.607842 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.607646    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-home\") pod \"54ab4e86-c68b-4fec-ba99-f8644088d920\" (UID: \"54ab4e86-c68b-4fec-ba99-f8644088d920\") "
Sep 16 19:27:17.608046 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.607865    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54ab4e86-c68b-4fec-ba99-f8644088d920" (UID: "54ab4e86-c68b-4fec-ba99-f8644088d920"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:17.608046 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.607967    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-model-cache" (OuterVolumeSpecName: "model-cache") pod "54ab4e86-c68b-4fec-ba99-f8644088d920" (UID: "54ab4e86-c68b-4fec-ba99-f8644088d920"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:17.608046 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.608024    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:17.608187 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.608037    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-home" (OuterVolumeSpecName: "home") pod "54ab4e86-c68b-4fec-ba99-f8644088d920" (UID: "54ab4e86-c68b-4fec-ba99-f8644088d920"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:17.610369 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.610345    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-dshm" (OuterVolumeSpecName: "dshm") pod "54ab4e86-c68b-4fec-ba99-f8644088d920" (UID: "54ab4e86-c68b-4fec-ba99-f8644088d920"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:17.610463 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.610365    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ab4e86-c68b-4fec-ba99-f8644088d920-kube-api-access-gt2fc" (OuterVolumeSpecName: "kube-api-access-gt2fc") pod "54ab4e86-c68b-4fec-ba99-f8644088d920" (UID: "54ab4e86-c68b-4fec-ba99-f8644088d920"). InnerVolumeSpecName "kube-api-access-gt2fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:17.610463 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.610446    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ab4e86-c68b-4fec-ba99-f8644088d920-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54ab4e86-c68b-4fec-ba99-f8644088d920" (UID: "54ab4e86-c68b-4fec-ba99-f8644088d920"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:17.708889 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.708846    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:17.708889 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.708882    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt2fc\" (UniqueName: \"kubernetes.io/projected/54ab4e86-c68b-4fec-ba99-f8644088d920-kube-api-access-gt2fc\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:17.708889 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.708893    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:17.709127 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.708902    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54ab4e86-c68b-4fec-ba99-f8644088d920-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:17.709127 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.708910    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54ab4e86-c68b-4fec-ba99-f8644088d920-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:17.822812 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:17.822756    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6" podUID="8beb2f94-9bf0-4136-927f-7475898a1e92" containerName="main" containerID="cri-o://040c8fa5b0f547629d1fa67e9e45682727c4b630044989acc9f7bd54dd98b5d1" gracePeriod=45
Sep 16 19:27:18.062225 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.062194    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:27:18.112401 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.112295    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx4vs\" (UniqueName: \"kubernetes.io/projected/8beb2f94-9bf0-4136-927f-7475898a1e92-kube-api-access-qx4vs\") pod \"8beb2f94-9bf0-4136-927f-7475898a1e92\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") "
Sep 16 19:27:18.112401 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.112372    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-tmp-dir\") pod \"8beb2f94-9bf0-4136-927f-7475898a1e92\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") "
Sep 16 19:27:18.112792 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.112456    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-dshm\") pod \"8beb2f94-9bf0-4136-927f-7475898a1e92\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") "
Sep 16 19:27:18.112792 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.112484    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8beb2f94-9bf0-4136-927f-7475898a1e92-tls-certs\") pod \"8beb2f94-9bf0-4136-927f-7475898a1e92\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") "
Sep 16 19:27:18.112792 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.112500    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-model-cache\") pod \"8beb2f94-9bf0-4136-927f-7475898a1e92\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") "
Sep 16 19:27:18.112792 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.112524    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-home\") pod \"8beb2f94-9bf0-4136-927f-7475898a1e92\" (UID: \"8beb2f94-9bf0-4136-927f-7475898a1e92\") "
Sep 16 19:27:18.112792 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.112725    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8beb2f94-9bf0-4136-927f-7475898a1e92" (UID: "8beb2f94-9bf0-4136-927f-7475898a1e92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:18.113038 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.112854    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:18.113038 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.112866    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-model-cache" (OuterVolumeSpecName: "model-cache") pod "8beb2f94-9bf0-4136-927f-7475898a1e92" (UID: "8beb2f94-9bf0-4136-927f-7475898a1e92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:18.113038 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.112882    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-home" (OuterVolumeSpecName: "home") pod "8beb2f94-9bf0-4136-927f-7475898a1e92" (UID: "8beb2f94-9bf0-4136-927f-7475898a1e92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:18.114966 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.114943    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-dshm" (OuterVolumeSpecName: "dshm") pod "8beb2f94-9bf0-4136-927f-7475898a1e92" (UID: "8beb2f94-9bf0-4136-927f-7475898a1e92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:18.115029 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.115002    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8beb2f94-9bf0-4136-927f-7475898a1e92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8beb2f94-9bf0-4136-927f-7475898a1e92" (UID: "8beb2f94-9bf0-4136-927f-7475898a1e92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:18.115029 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.115009    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8beb2f94-9bf0-4136-927f-7475898a1e92-kube-api-access-qx4vs" (OuterVolumeSpecName: "kube-api-access-qx4vs") pod "8beb2f94-9bf0-4136-927f-7475898a1e92" (UID: "8beb2f94-9bf0-4136-927f-7475898a1e92"). InnerVolumeSpecName "kube-api-access-qx4vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:18.213485 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.213440    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:18.213485 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.213481    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8beb2f94-9bf0-4136-927f-7475898a1e92-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:18.213766 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.213497    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:18.213766 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.213510    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8beb2f94-9bf0-4136-927f-7475898a1e92-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:18.213766 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.213523    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx4vs\" (UniqueName: \"kubernetes.io/projected/8beb2f94-9bf0-4136-927f-7475898a1e92-kube-api-access-qx4vs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:18.217161 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.217121    2582 generic.go:356] "Generic (PLEG): container finished" podID="8beb2f94-9bf0-4136-927f-7475898a1e92" containerID="040c8fa5b0f547629d1fa67e9e45682727c4b630044989acc9f7bd54dd98b5d1" exitCode=0
Sep 16 19:27:18.217313 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.217178    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"
Sep 16 19:27:18.217313 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.217177    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6" event={"ID":"8beb2f94-9bf0-4136-927f-7475898a1e92","Type":"ContainerDied","Data":"040c8fa5b0f547629d1fa67e9e45682727c4b630044989acc9f7bd54dd98b5d1"}
Sep 16 19:27:18.217313 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.217250    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6" event={"ID":"8beb2f94-9bf0-4136-927f-7475898a1e92","Type":"ContainerDied","Data":"b9c54a7da695eafccb5ce66d6a997f3e045b72ff3124fa533441b012da1cec0c"}
Sep 16 19:27:18.217313 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.217281    2582 scope.go:117] "RemoveContainer" containerID="040c8fa5b0f547629d1fa67e9e45682727c4b630044989acc9f7bd54dd98b5d1"
Sep 16 19:27:18.219041 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.219008    2582 generic.go:356] "Generic (PLEG): container finished" podID="54ab4e86-c68b-4fec-ba99-f8644088d920" containerID="4797c9d78ec8e0f2c1b655e0ee7e0f3752367ad68e375662c7ec696baf0f16b0" exitCode=0
Sep 16 19:27:18.219171 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.219046    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx" event={"ID":"54ab4e86-c68b-4fec-ba99-f8644088d920","Type":"ContainerDied","Data":"4797c9d78ec8e0f2c1b655e0ee7e0f3752367ad68e375662c7ec696baf0f16b0"}
Sep 16 19:27:18.219171 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.219069    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"
Sep 16 19:27:18.219171 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.219092    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx" event={"ID":"54ab4e86-c68b-4fec-ba99-f8644088d920","Type":"ContainerDied","Data":"17e53121748d5e39b57896cf41b521e72e369fbc59d07e84efff04a74b1abec9"}
Sep 16 19:27:18.229147 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.229027    2582 scope.go:117] "RemoveContainer" containerID="040c8fa5b0f547629d1fa67e9e45682727c4b630044989acc9f7bd54dd98b5d1"
Sep 16 19:27:18.229418 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:27:18.229399    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040c8fa5b0f547629d1fa67e9e45682727c4b630044989acc9f7bd54dd98b5d1\": container with ID starting with 040c8fa5b0f547629d1fa67e9e45682727c4b630044989acc9f7bd54dd98b5d1 not found: ID does not exist" containerID="040c8fa5b0f547629d1fa67e9e45682727c4b630044989acc9f7bd54dd98b5d1"
Sep 16 19:27:18.229551 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.229432    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040c8fa5b0f547629d1fa67e9e45682727c4b630044989acc9f7bd54dd98b5d1"} err="failed to get container status \"040c8fa5b0f547629d1fa67e9e45682727c4b630044989acc9f7bd54dd98b5d1\": rpc error: code = NotFound desc = could not find container \"040c8fa5b0f547629d1fa67e9e45682727c4b630044989acc9f7bd54dd98b5d1\": container with ID starting with 040c8fa5b0f547629d1fa67e9e45682727c4b630044989acc9f7bd54dd98b5d1 not found: ID does not exist"
Sep 16 19:27:18.229551 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.229450    2582 scope.go:117] "RemoveContainer" containerID="4797c9d78ec8e0f2c1b655e0ee7e0f3752367ad68e375662c7ec696baf0f16b0"
Sep 16 19:27:18.239938 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.239917    2582 scope.go:117] "RemoveContainer" containerID="4797c9d78ec8e0f2c1b655e0ee7e0f3752367ad68e375662c7ec696baf0f16b0"
Sep 16 19:27:18.240287 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:27:18.240254    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4797c9d78ec8e0f2c1b655e0ee7e0f3752367ad68e375662c7ec696baf0f16b0\": container with ID starting with 4797c9d78ec8e0f2c1b655e0ee7e0f3752367ad68e375662c7ec696baf0f16b0 not found: ID does not exist" containerID="4797c9d78ec8e0f2c1b655e0ee7e0f3752367ad68e375662c7ec696baf0f16b0"
Sep 16 19:27:18.240392 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.240299    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4797c9d78ec8e0f2c1b655e0ee7e0f3752367ad68e375662c7ec696baf0f16b0"} err="failed to get container status \"4797c9d78ec8e0f2c1b655e0ee7e0f3752367ad68e375662c7ec696baf0f16b0\": rpc error: code = NotFound desc = could not find container \"4797c9d78ec8e0f2c1b655e0ee7e0f3752367ad68e375662c7ec696baf0f16b0\": container with ID starting with 4797c9d78ec8e0f2c1b655e0ee7e0f3752367ad68e375662c7ec696baf0f16b0 not found: ID does not exist"
Sep 16 19:27:18.242959 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.242931    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"]
Sep 16 19:27:18.246997 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.246973    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-8s9q6"]
Sep 16 19:27:18.258859 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.258832    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"]
Sep 16 19:27:18.263110 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.263085    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-gg7sx"]
Sep 16 19:27:18.714078 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.714028    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54ab4e86-c68b-4fec-ba99-f8644088d920" path="/var/lib/kubelet/pods/54ab4e86-c68b-4fec-ba99-f8644088d920/volumes"
Sep 16 19:27:18.714496 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:18.714482    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8beb2f94-9bf0-4136-927f-7475898a1e92" path="/var/lib/kubelet/pods/8beb2f94-9bf0-4136-927f-7475898a1e92/volumes"
Sep 16 19:27:21.476378 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:21.476330    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql" podUID="ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:27:22.433108 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:22.433059    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:22.446106 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:22.446074    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:23.258022 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:23.257979    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:27:28.886603 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:28.886533    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql" podUID="ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c" containerName="main" containerID="cri-o://82b16eb4999a5b21c509bf58a7b9c8da7948c776f47a55e3a6621aa63d302e93" gracePeriod=45
Sep 16 19:27:31.475815 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:31.475770    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql" podUID="ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:27:41.475844 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:41.475752    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql" podUID="ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:27:41.476371 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:41.475876    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"
Sep 16 19:27:43.021681 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:43.021634    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:48.545258 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:48.545200    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:48.625542 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:48.625499    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:51.476030 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:51.475989    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql" podUID="ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:27:55.013087 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.013041    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"]
Sep 16 19:27:55.013661 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.013428    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ab4e86-c68b-4fec-ba99-f8644088d920" containerName="main"
Sep 16 19:27:55.013661 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.013440    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ab4e86-c68b-4fec-ba99-f8644088d920" containerName="main"
Sep 16 19:27:55.013661 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.013450    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8beb2f94-9bf0-4136-927f-7475898a1e92" containerName="main"
Sep 16 19:27:55.013661 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.013455    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8beb2f94-9bf0-4136-927f-7475898a1e92" containerName="main"
Sep 16 19:27:55.013661 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.013531    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ab4e86-c68b-4fec-ba99-f8644088d920" containerName="main"
Sep 16 19:27:55.013661 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.013540    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8beb2f94-9bf0-4136-927f-7475898a1e92" containerName="main"
Sep 16 19:27:55.016657 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.016631    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.022797 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.022765    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 16 19:27:55.040986 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.040948    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"]
Sep 16 19:27:55.062662 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.062628    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:55.068803 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.068766    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-model-cache\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.068934 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.068829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-tmp-dir\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.068934 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.068871    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-home\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.069021 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.068935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rddrf\" (UniqueName: \"kubernetes.io/projected/fc84b6f8-d896-479a-80e5-705ce6000c98-kube-api-access-rddrf\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.069021 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.068974    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc84b6f8-d896-479a-80e5-705ce6000c98-tls-certs\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.069021 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.068996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-dshm\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.169496 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.169438    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-home\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.169496 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.169500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rddrf\" (UniqueName: \"kubernetes.io/projected/fc84b6f8-d896-479a-80e5-705ce6000c98-kube-api-access-rddrf\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.169826 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.169525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc84b6f8-d896-479a-80e5-705ce6000c98-tls-certs\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.169826 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.169546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-dshm\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.169826 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.169623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-model-cache\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.169826 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.169668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-tmp-dir\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.170061 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.170035    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-home\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.170117 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.170048    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-tmp-dir\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.170117 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.170102    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-model-cache\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.172250 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.172227    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-dshm\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.172538 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.172519    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc84b6f8-d896-479a-80e5-705ce6000c98-tls-certs\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.178718 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.178698    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rddrf\" (UniqueName: \"kubernetes.io/projected/fc84b6f8-d896-479a-80e5-705ce6000c98-kube-api-access-rddrf\") pod \"late-v2-kserve-86766546f-2qbs7\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.327739 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.327692    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:55.614086 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:55.613978    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"]
Sep 16 19:27:55.618217 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:27:55.618178    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc84b6f8_d896_479a_80e5_705ce6000c98.slice/crio-eac6d940d92515e3200019fb58e66065c96a6902be5e6ae8f250bd30efd2d91f WatchSource:0}: Error finding container eac6d940d92515e3200019fb58e66065c96a6902be5e6ae8f250bd30efd2d91f: Status 404 returned error can't find the container with id eac6d940d92515e3200019fb58e66065c96a6902be5e6ae8f250bd30efd2d91f
Sep 16 19:27:56.418803 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:56.418745    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7" event={"ID":"fc84b6f8-d896-479a-80e5-705ce6000c98","Type":"ContainerStarted","Data":"0658f53e92a9f7b3a39c2e1905fd724ed9a866e1f2df112e2cd58ad109ee5d22"}
Sep 16 19:27:56.419242 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:56.418821    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7" event={"ID":"fc84b6f8-d896-479a-80e5-705ce6000c98","Type":"ContainerStarted","Data":"eac6d940d92515e3200019fb58e66065c96a6902be5e6ae8f250bd30efd2d91f"}
Sep 16 19:27:56.440452 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:56.440390    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7" podStartSLOduration=2.440369773 podStartE2EDuration="2.440369773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:56.438875189 +0000 UTC m=+2534.269483745" watchObservedRunningTime="2026-09-16 19:27:56.440369773 +0000 UTC m=+2534.270978360"
Sep 16 19:27:57.328093 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:57.328050    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:27:59.143007 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.142984    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"
Sep 16 19:27:59.205615 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.205542    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c-tls-certs\") pod \"ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c\" (UID: \"ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c\") "
Sep 16 19:27:59.205780 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.205630    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq79q\" (UniqueName: \"kubernetes.io/projected/ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c-kube-api-access-vq79q\") pod \"ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c\" (UID: \"ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c\") "
Sep 16 19:27:59.207751 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.207721    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c" (UID: "ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:59.207841 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.207818    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c-kube-api-access-vq79q" (OuterVolumeSpecName: "kube-api-access-vq79q") pod "ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c" (UID: "ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c"). InnerVolumeSpecName "kube-api-access-vq79q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:59.306941 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.306919    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:59.306941 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.306942    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq79q\" (UniqueName: \"kubernetes.io/projected/ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c-kube-api-access-vq79q\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:27:59.433539 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.433513    2582 generic.go:356] "Generic (PLEG): container finished" podID="ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c" containerID="82b16eb4999a5b21c509bf58a7b9c8da7948c776f47a55e3a6621aa63d302e93" exitCode=0
Sep 16 19:27:59.433655 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.433554    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"
Sep 16 19:27:59.433655 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.433610    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql" event={"ID":"ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c","Type":"ContainerDied","Data":"82b16eb4999a5b21c509bf58a7b9c8da7948c776f47a55e3a6621aa63d302e93"}
Sep 16 19:27:59.433737 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.433661    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql" event={"ID":"ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c","Type":"ContainerDied","Data":"58eef4999f120c14237714638d4aa1b88a5db611709df7f424046622c6d515db"}
Sep 16 19:27:59.433737 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.433682    2582 scope.go:117] "RemoveContainer" containerID="82b16eb4999a5b21c509bf58a7b9c8da7948c776f47a55e3a6621aa63d302e93"
Sep 16 19:27:59.442371 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.442352    2582 scope.go:117] "RemoveContainer" containerID="82b16eb4999a5b21c509bf58a7b9c8da7948c776f47a55e3a6621aa63d302e93"
Sep 16 19:27:59.442660 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:27:59.442639    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b16eb4999a5b21c509bf58a7b9c8da7948c776f47a55e3a6621aa63d302e93\": container with ID starting with 82b16eb4999a5b21c509bf58a7b9c8da7948c776f47a55e3a6621aa63d302e93 not found: ID does not exist" containerID="82b16eb4999a5b21c509bf58a7b9c8da7948c776f47a55e3a6621aa63d302e93"
Sep 16 19:27:59.442718 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.442669    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b16eb4999a5b21c509bf58a7b9c8da7948c776f47a55e3a6621aa63d302e93"} err="failed to get container status \"82b16eb4999a5b21c509bf58a7b9c8da7948c776f47a55e3a6621aa63d302e93\": rpc error: code = NotFound desc = could not find container \"82b16eb4999a5b21c509bf58a7b9c8da7948c776f47a55e3a6621aa63d302e93\": container with ID starting with 82b16eb4999a5b21c509bf58a7b9c8da7948c776f47a55e3a6621aa63d302e93 not found: ID does not exist"
Sep 16 19:27:59.456827 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.456771    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"]
Sep 16 19:27:59.462830 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.462806    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehmbql"]
Sep 16 19:27:59.813539 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.813514    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:59.910236 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:27:59.910212    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:00.704543 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:00.704504    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c" path="/var/lib/kubelet/pods/ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c/volumes"
Sep 16 19:28:05.328685 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:05.328635    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:28:05.341911 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:05.341879    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:28:05.473341 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:05.473311    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:28:06.185816 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:06.185755    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:11.324938 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.324909    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:11.571394 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.571354    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"]
Sep 16 19:28:11.571828 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.571809    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c" containerName="main"
Sep 16 19:28:11.571932 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.571830    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c" containerName="main"
Sep 16 19:28:11.571990 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.571938    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddaf99db-2b0c-4b0f-abc3-2e22824d3a6c" containerName="main"
Sep 16 19:28:11.576332 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.576274    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"
Sep 16 19:28:11.581064 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.581036    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-djk5h\""
Sep 16 19:28:11.581064 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.581066    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-r5s47\""
Sep 16 19:28:11.581247 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.581084    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:28:11.581247 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.581123    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:28:11.581247 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.581093    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 19:28:11.584385 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.584365    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"]
Sep 16 19:28:11.598858 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.598838    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:11.715713 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.715634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqz6f\" (UniqueName: \"kubernetes.io/projected/8edba542-89ca-4b0c-a28d-4fafeea97647-kube-api-access-zqz6f\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p\" (UID: \"8edba542-89ca-4b0c-a28d-4fafeea97647\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"
Sep 16 19:28:11.715713 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.715679    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8edba542-89ca-4b0c-a28d-4fafeea97647-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p\" (UID: \"8edba542-89ca-4b0c-a28d-4fafeea97647\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"
Sep 16 19:28:11.816384 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.816353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqz6f\" (UniqueName: \"kubernetes.io/projected/8edba542-89ca-4b0c-a28d-4fafeea97647-kube-api-access-zqz6f\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p\" (UID: \"8edba542-89ca-4b0c-a28d-4fafeea97647\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"
Sep 16 19:28:11.816460 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.816415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8edba542-89ca-4b0c-a28d-4fafeea97647-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p\" (UID: \"8edba542-89ca-4b0c-a28d-4fafeea97647\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"
Sep 16 19:28:11.818897 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.818877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8edba542-89ca-4b0c-a28d-4fafeea97647-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p\" (UID: \"8edba542-89ca-4b0c-a28d-4fafeea97647\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"
Sep 16 19:28:11.826138 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.826119    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqz6f\" (UniqueName: \"kubernetes.io/projected/8edba542-89ca-4b0c-a28d-4fafeea97647-kube-api-access-zqz6f\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p\" (UID: \"8edba542-89ca-4b0c-a28d-4fafeea97647\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"
Sep 16 19:28:11.887209 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:11.887096    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"
Sep 16 19:28:12.160166 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:12.160141    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"]
Sep 16 19:28:12.162349 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:28:12.162317    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edba542_89ca_4b0c_a28d_4fafeea97647.slice/crio-193ed0e44f21ce7d839b2a2a3df70b15466695777dfa2afa35cd1aaec0aefc2e WatchSource:0}: Error finding container 193ed0e44f21ce7d839b2a2a3df70b15466695777dfa2afa35cd1aaec0aefc2e: Status 404 returned error can't find the container with id 193ed0e44f21ce7d839b2a2a3df70b15466695777dfa2afa35cd1aaec0aefc2e
Sep 16 19:28:12.255810 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:12.255761    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:12.357720 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:12.357684    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:12.498756 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:12.498676    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p" event={"ID":"8edba542-89ca-4b0c-a28d-4fafeea97647","Type":"ContainerStarted","Data":"ec2a64afaa39b4a81205fa6bb9801461921f8c73fdf1b281f988e22e4d33e0b2"}
Sep 16 19:28:12.498756 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:12.498734    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p" event={"ID":"8edba542-89ca-4b0c-a28d-4fafeea97647","Type":"ContainerStarted","Data":"193ed0e44f21ce7d839b2a2a3df70b15466695777dfa2afa35cd1aaec0aefc2e"}
Sep 16 19:28:12.498952 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:12.498812    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"
Sep 16 19:28:40.867349 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:40.867311    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:40.969866 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:40.969827    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:41.071688 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:41.071654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:43.505101 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:43.505069    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"
Sep 16 19:28:43.525410 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:43.525355    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p" podStartSLOduration=32.525337224 podStartE2EDuration="32.525337224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:12.522513764 +0000 UTC m=+2550.353122322" watchObservedRunningTime="2026-09-16 19:28:43.525337224 +0000 UTC m=+2581.355945768"
Sep 16 19:28:44.600431 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:44.600390    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:45.853615 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:45.853552    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"]
Sep 16 19:28:45.859687 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:45.859653    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"]
Sep 16 19:28:45.920741 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:45.920697    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:45.983588 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:45.983529    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:47.022041 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:28:47.022007    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:00.875530 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:00.875481    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g" podUID="1b1fb9c4-6cdb-4dde-b911-49d4b467e40b" containerName="main" containerID="cri-o://542f749d212021577d06d2ff5161f74a5c908c573820d6e1d1cec90f4fcead94" gracePeriod=45
Sep 16 19:29:00.881711 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:00.881673    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7" podUID="fc84b6f8-d896-479a-80e5-705ce6000c98" containerName="main" containerID="cri-o://0658f53e92a9f7b3a39c2e1905fd724ed9a866e1f2df112e2cd58ad109ee5d22" gracePeriod=45
Sep 16 19:29:01.130691 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.130620    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:29:01.158012 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.157984    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:29:01.259503 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.259455    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rddrf\" (UniqueName: \"kubernetes.io/projected/fc84b6f8-d896-479a-80e5-705ce6000c98-kube-api-access-rddrf\") pod \"fc84b6f8-d896-479a-80e5-705ce6000c98\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") "
Sep 16 19:29:01.259751 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.259517    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-model-cache\") pod \"fc84b6f8-d896-479a-80e5-705ce6000c98\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") "
Sep 16 19:29:01.259751 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.259563    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-dshm\") pod \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") "
Sep 16 19:29:01.259751 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.259615    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdtdj\" (UniqueName: \"kubernetes.io/projected/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-kube-api-access-qdtdj\") pod \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") "
Sep 16 19:29:01.259751 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.259638    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-home\") pod \"fc84b6f8-d896-479a-80e5-705ce6000c98\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") "
Sep 16 19:29:01.259751 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.259675    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-model-cache\") pod \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") "
Sep 16 19:29:01.259751 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.259705    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-tmp-dir\") pod \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") "
Sep 16 19:29:01.259751 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.259742    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-tmp-dir\") pod \"fc84b6f8-d896-479a-80e5-705ce6000c98\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") "
Sep 16 19:29:01.260192 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.259765    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-tls-certs\") pod \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") "
Sep 16 19:29:01.260192 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.259818    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-home\") pod \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\" (UID: \"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b\") "
Sep 16 19:29:01.260192 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.259827    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc84b6f8-d896-479a-80e5-705ce6000c98" (UID: "fc84b6f8-d896-479a-80e5-705ce6000c98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:01.260192 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.259857    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc84b6f8-d896-479a-80e5-705ce6000c98-tls-certs\") pod \"fc84b6f8-d896-479a-80e5-705ce6000c98\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") "
Sep 16 19:29:01.260192 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.259882    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-dshm\") pod \"fc84b6f8-d896-479a-80e5-705ce6000c98\" (UID: \"fc84b6f8-d896-479a-80e5-705ce6000c98\") "
Sep 16 19:29:01.260192 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.259952    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-home" (OuterVolumeSpecName: "home") pod "fc84b6f8-d896-479a-80e5-705ce6000c98" (UID: "fc84b6f8-d896-479a-80e5-705ce6000c98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:01.260192 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.260046    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc84b6f8-d896-479a-80e5-705ce6000c98" (UID: "fc84b6f8-d896-479a-80e5-705ce6000c98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:01.260192 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.260104    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b1fb9c4-6cdb-4dde-b911-49d4b467e40b" (UID: "1b1fb9c4-6cdb-4dde-b911-49d4b467e40b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:01.260814 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.260226    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b1fb9c4-6cdb-4dde-b911-49d4b467e40b" (UID: "1b1fb9c4-6cdb-4dde-b911-49d4b467e40b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:01.260814 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.260241    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-home" (OuterVolumeSpecName: "home") pod "1b1fb9c4-6cdb-4dde-b911-49d4b467e40b" (UID: "1b1fb9c4-6cdb-4dde-b911-49d4b467e40b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:01.260814 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.260299    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:29:01.260814 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.260315    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:29:01.260814 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.260328    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:29:01.260814 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.260341    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:29:01.262346 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.262307    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-kube-api-access-qdtdj" (OuterVolumeSpecName: "kube-api-access-qdtdj") pod "1b1fb9c4-6cdb-4dde-b911-49d4b467e40b" (UID: "1b1fb9c4-6cdb-4dde-b911-49d4b467e40b"). InnerVolumeSpecName "kube-api-access-qdtdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:01.262346 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.262323    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-dshm" (OuterVolumeSpecName: "dshm") pod "fc84b6f8-d896-479a-80e5-705ce6000c98" (UID: "fc84b6f8-d896-479a-80e5-705ce6000c98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:01.262510 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.262478    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-dshm" (OuterVolumeSpecName: "dshm") pod "1b1fb9c4-6cdb-4dde-b911-49d4b467e40b" (UID: "1b1fb9c4-6cdb-4dde-b911-49d4b467e40b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:01.262510 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.262487    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc84b6f8-d896-479a-80e5-705ce6000c98-kube-api-access-rddrf" (OuterVolumeSpecName: "kube-api-access-rddrf") pod "fc84b6f8-d896-479a-80e5-705ce6000c98" (UID: "fc84b6f8-d896-479a-80e5-705ce6000c98"). InnerVolumeSpecName "kube-api-access-rddrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:01.263044 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.263018    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b1fb9c4-6cdb-4dde-b911-49d4b467e40b" (UID: "1b1fb9c4-6cdb-4dde-b911-49d4b467e40b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:01.263110 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.263044    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc84b6f8-d896-479a-80e5-705ce6000c98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc84b6f8-d896-479a-80e5-705ce6000c98" (UID: "fc84b6f8-d896-479a-80e5-705ce6000c98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:01.361657 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.361613    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rddrf\" (UniqueName: \"kubernetes.io/projected/fc84b6f8-d896-479a-80e5-705ce6000c98-kube-api-access-rddrf\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:29:01.361657 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.361648    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:29:01.361657 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.361659    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdtdj\" (UniqueName: \"kubernetes.io/projected/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-kube-api-access-qdtdj\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:29:01.361657 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.361668    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:29:01.361942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.361678    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:29:01.361942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.361685    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:29:01.361942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.361692    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc84b6f8-d896-479a-80e5-705ce6000c98-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:29:01.361942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.361700    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc84b6f8-d896-479a-80e5-705ce6000c98-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:29:01.733143 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.733101    2582 generic.go:356] "Generic (PLEG): container finished" podID="fc84b6f8-d896-479a-80e5-705ce6000c98" containerID="0658f53e92a9f7b3a39c2e1905fd724ed9a866e1f2df112e2cd58ad109ee5d22" exitCode=0
Sep 16 19:29:01.733356 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.733151    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"
Sep 16 19:29:01.733356 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.733197    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7" event={"ID":"fc84b6f8-d896-479a-80e5-705ce6000c98","Type":"ContainerDied","Data":"0658f53e92a9f7b3a39c2e1905fd724ed9a866e1f2df112e2cd58ad109ee5d22"}
Sep 16 19:29:01.733356 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.733261    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7" event={"ID":"fc84b6f8-d896-479a-80e5-705ce6000c98","Type":"ContainerDied","Data":"eac6d940d92515e3200019fb58e66065c96a6902be5e6ae8f250bd30efd2d91f"}
Sep 16 19:29:01.733356 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.733292    2582 scope.go:117] "RemoveContainer" containerID="0658f53e92a9f7b3a39c2e1905fd724ed9a866e1f2df112e2cd58ad109ee5d22"
Sep 16 19:29:01.734671 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.734651    2582 generic.go:356] "Generic (PLEG): container finished" podID="1b1fb9c4-6cdb-4dde-b911-49d4b467e40b" containerID="542f749d212021577d06d2ff5161f74a5c908c573820d6e1d1cec90f4fcead94" exitCode=0
Sep 16 19:29:01.734773 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.734701    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"
Sep 16 19:29:01.734773 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.734723    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g" event={"ID":"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b","Type":"ContainerDied","Data":"542f749d212021577d06d2ff5161f74a5c908c573820d6e1d1cec90f4fcead94"}
Sep 16 19:29:01.734773 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.734754    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g" event={"ID":"1b1fb9c4-6cdb-4dde-b911-49d4b467e40b","Type":"ContainerDied","Data":"9afdc5b0155654b94f51ba46c15c0f1f1df3bab3871c440194da10dde2c84ac6"}
Sep 16 19:29:01.742465 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.742443    2582 scope.go:117] "RemoveContainer" containerID="0658f53e92a9f7b3a39c2e1905fd724ed9a866e1f2df112e2cd58ad109ee5d22"
Sep 16 19:29:01.742792 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:29:01.742767    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0658f53e92a9f7b3a39c2e1905fd724ed9a866e1f2df112e2cd58ad109ee5d22\": container with ID starting with 0658f53e92a9f7b3a39c2e1905fd724ed9a866e1f2df112e2cd58ad109ee5d22 not found: ID does not exist" containerID="0658f53e92a9f7b3a39c2e1905fd724ed9a866e1f2df112e2cd58ad109ee5d22"
Sep 16 19:29:01.742891 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.742800    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0658f53e92a9f7b3a39c2e1905fd724ed9a866e1f2df112e2cd58ad109ee5d22"} err="failed to get container status \"0658f53e92a9f7b3a39c2e1905fd724ed9a866e1f2df112e2cd58ad109ee5d22\": rpc error: code = NotFound desc = could not find container \"0658f53e92a9f7b3a39c2e1905fd724ed9a866e1f2df112e2cd58ad109ee5d22\": container with ID starting with 0658f53e92a9f7b3a39c2e1905fd724ed9a866e1f2df112e2cd58ad109ee5d22 not found: ID does not exist"
Sep 16 19:29:01.742891 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.742817    2582 scope.go:117] "RemoveContainer" containerID="542f749d212021577d06d2ff5161f74a5c908c573820d6e1d1cec90f4fcead94"
Sep 16 19:29:01.751193 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.751175    2582 scope.go:117] "RemoveContainer" containerID="542f749d212021577d06d2ff5161f74a5c908c573820d6e1d1cec90f4fcead94"
Sep 16 19:29:01.751477 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:29:01.751462    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"542f749d212021577d06d2ff5161f74a5c908c573820d6e1d1cec90f4fcead94\": container with ID starting with 542f749d212021577d06d2ff5161f74a5c908c573820d6e1d1cec90f4fcead94 not found: ID does not exist" containerID="542f749d212021577d06d2ff5161f74a5c908c573820d6e1d1cec90f4fcead94"
Sep 16 19:29:01.751525 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.751485    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"542f749d212021577d06d2ff5161f74a5c908c573820d6e1d1cec90f4fcead94"} err="failed to get container status \"542f749d212021577d06d2ff5161f74a5c908c573820d6e1d1cec90f4fcead94\": rpc error: code = NotFound desc = could not find container \"542f749d212021577d06d2ff5161f74a5c908c573820d6e1d1cec90f4fcead94\": container with ID starting with 542f749d212021577d06d2ff5161f74a5c908c573820d6e1d1cec90f4fcead94 not found: ID does not exist"
Sep 16 19:29:01.760673 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.760648    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"]
Sep 16 19:29:01.763824 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.763802    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-2qbs7"]
Sep 16 19:29:01.775857 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.775829    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"]
Sep 16 19:29:01.777752 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:01.777727    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-2655g"]
Sep 16 19:29:02.703979 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:02.703941    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b1fb9c4-6cdb-4dde-b911-49d4b467e40b" path="/var/lib/kubelet/pods/1b1fb9c4-6cdb-4dde-b911-49d4b467e40b/volumes"
Sep 16 19:29:02.704490 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:02.704469    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc84b6f8-d896-479a-80e5-705ce6000c98" path="/var/lib/kubelet/pods/fc84b6f8-d896-479a-80e5-705ce6000c98/volumes"
Sep 16 19:29:11.111124 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.111016    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"]
Sep 16 19:29:11.111627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.111402    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1fb9c4-6cdb-4dde-b911-49d4b467e40b" containerName="main"
Sep 16 19:29:11.111627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.111416    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1fb9c4-6cdb-4dde-b911-49d4b467e40b" containerName="main"
Sep 16 19:29:11.111627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.111428    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc84b6f8-d896-479a-80e5-705ce6000c98" containerName="main"
Sep 16 19:29:11.111627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.111434    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc84b6f8-d896-479a-80e5-705ce6000c98" containerName="main"
Sep 16 19:29:11.111627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.111505    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc84b6f8-d896-479a-80e5-705ce6000c98" containerName="main"
Sep 16 19:29:11.111627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.111515    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b1fb9c4-6cdb-4dde-b911-49d4b467e40b" containerName="main"
Sep 16 19:29:11.115836 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.115810    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.118969 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.118943    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 16 19:29:11.120165 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.120141    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 16 19:29:11.120432 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.120410    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 16 19:29:11.120530 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.120451    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-4tr5d\""
Sep 16 19:29:11.126298 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.126273    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"]
Sep 16 19:29:11.134592 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.134527    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:11.250541 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.250496    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db7b07b-4df5-4a3e-b273-2f5698b8d317-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.250763 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.250554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-dshm\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.250763 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.250587    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fln6\" (UniqueName: \"kubernetes.io/projected/8db7b07b-4df5-4a3e-b273-2f5698b8d317-kube-api-access-4fln6\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.250763 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.250683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.250763 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.250726    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.250763 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.250755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-home\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.352029 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.351989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db7b07b-4df5-4a3e-b273-2f5698b8d317-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.352257 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.352048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-dshm\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.352257 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.352071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fln6\" (UniqueName: \"kubernetes.io/projected/8db7b07b-4df5-4a3e-b273-2f5698b8d317-kube-api-access-4fln6\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.352257 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.352103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.352257 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.352134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.352257 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.352169    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-home\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.352774 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.352743    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.353106 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.352979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-home\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.353106 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.353053    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.356608 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.355614    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-dshm\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.356608 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.355880    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db7b07b-4df5-4a3e-b273-2f5698b8d317-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.361925 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.361855    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fln6\" (UniqueName: \"kubernetes.io/projected/8db7b07b-4df5-4a3e-b273-2f5698b8d317-kube-api-access-4fln6\") pod \"delw-v1-kserve-6cf8cbb847-wz8cd\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.426889 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.426837    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:11.706112 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.706076    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"]
Sep 16 19:29:11.709118 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:29:11.709084    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db7b07b_4df5_4a3e_b273_2f5698b8d317.slice/crio-a978ffcabe755163132fdcca491732ef14248eb53ce025b0012cf4a9f063e453 WatchSource:0}: Error finding container a978ffcabe755163132fdcca491732ef14248eb53ce025b0012cf4a9f063e453: Status 404 returned error can't find the container with id a978ffcabe755163132fdcca491732ef14248eb53ce025b0012cf4a9f063e453
Sep 16 19:29:11.783367 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.783330    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd" event={"ID":"8db7b07b-4df5-4a3e-b273-2f5698b8d317","Type":"ContainerStarted","Data":"b04ce0051de9e0e399fd4dd6bac74c1ce15561b9e6f116eb34f1ed5c5c0c9772"}
Sep 16 19:29:11.783511 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.783390    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd" event={"ID":"8db7b07b-4df5-4a3e-b273-2f5698b8d317","Type":"ContainerStarted","Data":"a978ffcabe755163132fdcca491732ef14248eb53ce025b0012cf4a9f063e453"}
Sep 16 19:29:11.806897 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:11.806838    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd" podStartSLOduration=0.806819524 podStartE2EDuration="806.819524ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:11.805839488 +0000 UTC m=+2609.636448037" watchObservedRunningTime="2026-09-16 19:29:11.806819524 +0000 UTC m=+2609.637428068"
Sep 16 19:29:12.041876 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:12.041827    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:12.427557 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:12.427449    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:15.393285 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.393232    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"]
Sep 16 19:29:15.396730 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.396707    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.399642 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.399612    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 16 19:29:15.408594 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.408546    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"]
Sep 16 19:29:15.432031 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.432010    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:15.488448 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.488411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-home\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.488539 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.488461    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.488539 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.488503    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.488539 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.488535    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.488691 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.488594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.488691 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.488627    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k5rh\" (UniqueName: \"kubernetes.io/projected/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-kube-api-access-4k5rh\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.589802 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.589782    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.589895 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.589815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k5rh\" (UniqueName: \"kubernetes.io/projected/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-kube-api-access-4k5rh\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.589895 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.589884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-home\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.589975 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.589904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.589975 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.589929    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.589975 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.589964    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.590274 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.590255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.590359 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.590297    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-home\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.590359 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.590318    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.592344 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.592320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.592618 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.592600    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.598103 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.598082    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k5rh\" (UniqueName: \"kubernetes.io/projected/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-kube-api-access-4k5rh\") pod \"delw-v2-kserve-5b84cc7ddf-76vpw\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.708674 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.708619    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:15.980115 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:15.979959    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"]
Sep 16 19:29:15.982546 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:29:15.982514    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a0ab4_1ae1_4b97_8bb0_7ce87a1c63e6.slice/crio-3589c3125706576361d78d249936a23f70988a8bdd97f592aa1ae20f383dc577 WatchSource:0}: Error finding container 3589c3125706576361d78d249936a23f70988a8bdd97f592aa1ae20f383dc577: Status 404 returned error can't find the container with id 3589c3125706576361d78d249936a23f70988a8bdd97f592aa1ae20f383dc577
Sep 16 19:29:16.808984 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:16.808949    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw" event={"ID":"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6","Type":"ContainerStarted","Data":"ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad"}
Sep 16 19:29:16.809504 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:16.809012    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw" event={"ID":"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6","Type":"ContainerStarted","Data":"3589c3125706576361d78d249936a23f70988a8bdd97f592aa1ae20f383dc577"}
Sep 16 19:29:16.827090 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:16.827029    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw" podStartSLOduration=1.827004069 podStartE2EDuration="1.827004069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:16.826388858 +0000 UTC m=+2614.656997404" watchObservedRunningTime="2026-09-16 19:29:16.827004069 +0000 UTC m=+2614.657612608"
Sep 16 19:29:17.709494 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:17.709448    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:21.427502 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:21.427455    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:21.440968 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:21.440934    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:21.847137 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:21.847099    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:29:22.810033 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:22.809997    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:25.709773 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:25.709730    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:25.722311 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:25.722285    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:25.867893 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:25.867857    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:29:28.440252 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:28.440219    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:28.510924 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:28.510895    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:46.446551 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:46.446515    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:46.698736 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:29:46.698633    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:02.190998 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:02.190936    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:10.060830 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:10.060790    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:20.259871 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:20.259834    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:20.410810 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:20.410761    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:27.552878 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:27.552836    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"]
Sep 16 19:30:27.568606 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:27.568520    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:42.580124 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.579999    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw" podUID="3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6" containerName="main" containerID="cri-o://ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad" gracePeriod=45
Sep 16 19:30:42.847920 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.847819    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:30:42.964145 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.964096    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-dshm\") pod \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") "
Sep 16 19:30:42.964400 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.964196    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-tls-certs\") pod \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") "
Sep 16 19:30:42.964400 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.964221    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4k5rh\" (UniqueName: \"kubernetes.io/projected/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-kube-api-access-4k5rh\") pod \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") "
Sep 16 19:30:42.964400 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.964272    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-tmp-dir\") pod \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") "
Sep 16 19:30:42.964400 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.964294    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-model-cache\") pod \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") "
Sep 16 19:30:42.964674 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.964491    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-home\") pod \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\" (UID: \"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6\") "
Sep 16 19:30:42.964674 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.964603    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6" (UID: "3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:42.964674 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.964621    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6" (UID: "3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:42.964924 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.964898    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-home" (OuterVolumeSpecName: "home") pod "3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6" (UID: "3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:42.965116 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.964908    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:30:42.965382 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.965360    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:30:42.967079 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.967048    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-dshm" (OuterVolumeSpecName: "dshm") pod "3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6" (UID: "3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:42.967171 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.967128    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6" (UID: "3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:42.967213 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:42.967163    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-kube-api-access-4k5rh" (OuterVolumeSpecName: "kube-api-access-4k5rh") pod "3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6" (UID: "3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6"). InnerVolumeSpecName "kube-api-access-4k5rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:43.066893 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.066849    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:30:43.066893 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.066883    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:30:43.066893 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.066896    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k5rh\" (UniqueName: \"kubernetes.io/projected/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-kube-api-access-4k5rh\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:30:43.066893 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.066906    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:30:43.225351 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.225232    2582 generic.go:356] "Generic (PLEG): container finished" podID="3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6" containerID="ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad" exitCode=0
Sep 16 19:30:43.225351 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.225294    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"
Sep 16 19:30:43.225351 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.225325    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw" event={"ID":"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6","Type":"ContainerDied","Data":"ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad"}
Sep 16 19:30:43.225637 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.225402    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw" event={"ID":"3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6","Type":"ContainerDied","Data":"3589c3125706576361d78d249936a23f70988a8bdd97f592aa1ae20f383dc577"}
Sep 16 19:30:43.225637 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.225432    2582 scope.go:117] "RemoveContainer" containerID="ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad"
Sep 16 19:30:43.236485 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.236463    2582 scope.go:117] "RemoveContainer" containerID="ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad"
Sep 16 19:30:43.236940 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:30:43.236919    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad\": container with ID starting with ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad not found: ID does not exist" containerID="ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad"
Sep 16 19:30:43.237003 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.236953    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad"} err="failed to get container status \"ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad\": rpc error: code = NotFound desc = could not find container \"ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad\": container with ID starting with ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad not found: ID does not exist"
Sep 16 19:30:43.248776 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.248738    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"]
Sep 16 19:30:43.253150 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.253120    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-76vpw"]
Sep 16 19:30:43.560692 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.560659    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:43.595822 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.595775    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"]
Sep 16 19:30:43.669636 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.669600    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:43.761730 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:43.761686    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:44.704061 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:44.704015    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6" path="/var/lib/kubelet/pods/3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6/volumes"
Sep 16 19:30:48.639871 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:48.639803    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:49.109503 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.109459    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"]
Sep 16 19:30:49.110029 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.110007    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6" containerName="main"
Sep 16 19:30:49.110135 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.110033    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6" containerName="main"
Sep 16 19:30:49.110195 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.110155    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b7a0ab4-1ae1-4b97-8bb0-7ce87a1c63e6" containerName="main"
Sep 16 19:30:49.114826 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.114794    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.119333 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.119308    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 16 19:30:49.119624 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.119416    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 16 19:30:49.119624 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.119444    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-t5pzf\""
Sep 16 19:30:49.119856 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.119479    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Sep 16 19:30:49.121942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.121916    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"]
Sep 16 19:30:49.139058 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.139022    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:49.227598 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.227534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dljtk\" (UniqueName: \"kubernetes.io/projected/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-kube-api-access-dljtk\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.227817 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.227647    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-dshm\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.227817 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.227678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.227817 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.227769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-home\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.227817 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.227799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.227968 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.227828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.328915 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.328851    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dljtk\" (UniqueName: \"kubernetes.io/projected/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-kube-api-access-dljtk\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.329204 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.328933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-dshm\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.329204 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.328954    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.329204 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.329004    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-home\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.329204 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.329026    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.329204 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.329168    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.329532 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.329504    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-home\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.329641 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.329562    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.329699 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.329684    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.331901 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.331881    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-dshm\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.332086 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.332070    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.338554 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.338518    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dljtk\" (UniqueName: \"kubernetes.io/projected/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-kube-api-access-dljtk\") pod \"roll-v1-kserve-69cf8bcfc-zcjjm\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.427304 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.427175    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:49.713389 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:49.713363    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"]
Sep 16 19:30:49.720123 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:30:49.719818    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb62a1a_00a2_44ef_9a10_b0a53c2a13a8.slice/crio-4bf7c408d4b54516a131f2e3872810fc54c019e5adc427e6e1cfdb72801eae86 WatchSource:0}: Error finding container 4bf7c408d4b54516a131f2e3872810fc54c019e5adc427e6e1cfdb72801eae86: Status 404 returned error can't find the container with id 4bf7c408d4b54516a131f2e3872810fc54c019e5adc427e6e1cfdb72801eae86
Sep 16 19:30:50.268005 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:50.267950    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm" event={"ID":"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8","Type":"ContainerStarted","Data":"7ebdbbb3a10f1127d27bf3901d5041c8a365a4fe4d23c2587460444f7818da0c"}
Sep 16 19:30:50.268394 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:50.268047    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm" event={"ID":"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8","Type":"ContainerStarted","Data":"4bf7c408d4b54516a131f2e3872810fc54c019e5adc427e6e1cfdb72801eae86"}
Sep 16 19:30:50.287825 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:50.287745    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm" podStartSLOduration=1.287724409 podStartE2EDuration="1.287724409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:50.286754911 +0000 UTC m=+2708.117363457" watchObservedRunningTime="2026-09-16 19:30:50.287724409 +0000 UTC m=+2708.118332952"
Sep 16 19:30:50.427716 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:50.427666    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:51.596496 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:30:51.596456    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a0ab4_1ae1_4b97_8bb0_7ce87a1c63e6.slice/crio-ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:30:51.630076 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:30:51.630037    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a0ab4_1ae1_4b97_8bb0_7ce87a1c63e6.slice/crio-ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:30:56.711440 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:56.711400    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:56.769549 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:56.769508    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:58.617448 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:58.617392    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd" podUID="8db7b07b-4df5-4a3e-b273-2f5698b8d317" containerName="main" containerID="cri-o://b04ce0051de9e0e399fd4dd6bac74c1ce15561b9e6f116eb34f1ed5c5c0c9772" gracePeriod=45
Sep 16 19:30:58.877795 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:58.877721    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:30:58.926694 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:58.926656    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-home\") pod \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") "
Sep 16 19:30:58.926907 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:58.926726    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db7b07b-4df5-4a3e-b273-2f5698b8d317-tls-certs\") pod \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") "
Sep 16 19:30:58.926907 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:58.926765    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fln6\" (UniqueName: \"kubernetes.io/projected/8db7b07b-4df5-4a3e-b273-2f5698b8d317-kube-api-access-4fln6\") pod \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") "
Sep 16 19:30:58.926907 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:58.926804    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-model-cache\") pod \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") "
Sep 16 19:30:58.926907 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:58.926836    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-tmp-dir\") pod \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") "
Sep 16 19:30:58.927139 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:58.926980    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-home" (OuterVolumeSpecName: "home") pod "8db7b07b-4df5-4a3e-b273-2f5698b8d317" (UID: "8db7b07b-4df5-4a3e-b273-2f5698b8d317"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:58.927199 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:58.927133    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-model-cache" (OuterVolumeSpecName: "model-cache") pod "8db7b07b-4df5-4a3e-b273-2f5698b8d317" (UID: "8db7b07b-4df5-4a3e-b273-2f5698b8d317"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:58.927243 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:58.927205    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8db7b07b-4df5-4a3e-b273-2f5698b8d317" (UID: "8db7b07b-4df5-4a3e-b273-2f5698b8d317"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:58.927243 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:58.927227    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:30:58.929346 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:58.929321    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8db7b07b-4df5-4a3e-b273-2f5698b8d317-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8db7b07b-4df5-4a3e-b273-2f5698b8d317" (UID: "8db7b07b-4df5-4a3e-b273-2f5698b8d317"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:58.929476 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:58.929416    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db7b07b-4df5-4a3e-b273-2f5698b8d317-kube-api-access-4fln6" (OuterVolumeSpecName: "kube-api-access-4fln6") pod "8db7b07b-4df5-4a3e-b273-2f5698b8d317" (UID: "8db7b07b-4df5-4a3e-b273-2f5698b8d317"). InnerVolumeSpecName "kube-api-access-4fln6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:59.028396 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.028351    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-dshm\") pod \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\" (UID: \"8db7b07b-4df5-4a3e-b273-2f5698b8d317\") "
Sep 16 19:30:59.028674 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.028629    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db7b07b-4df5-4a3e-b273-2f5698b8d317-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:30:59.028674 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.028649    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fln6\" (UniqueName: \"kubernetes.io/projected/8db7b07b-4df5-4a3e-b273-2f5698b8d317-kube-api-access-4fln6\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:30:59.028674 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.028663    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:30:59.028814 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.028677    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:30:59.030864 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.030837    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-dshm" (OuterVolumeSpecName: "dshm") pod "8db7b07b-4df5-4a3e-b273-2f5698b8d317" (UID: "8db7b07b-4df5-4a3e-b273-2f5698b8d317"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:59.093513 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.093468    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"]
Sep 16 19:30:59.094112 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.094085    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db7b07b-4df5-4a3e-b273-2f5698b8d317" containerName="main"
Sep 16 19:30:59.094112 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.094112    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db7b07b-4df5-4a3e-b273-2f5698b8d317" containerName="main"
Sep 16 19:30:59.094317 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.094247    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db7b07b-4df5-4a3e-b273-2f5698b8d317" containerName="main"
Sep 16 19:30:59.097520 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.097489    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.100635 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.100607    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 16 19:30:59.109506 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.109420    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"]
Sep 16 19:30:59.129497 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.129379    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.129722 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.129604    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-dshm\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.129722 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.129693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-home\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.129817 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.129737    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.129817 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.129796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecd04dfe-4f45-4872-995a-060a166d25f8-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.129948 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.129923    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6nbs\" (UniqueName: \"kubernetes.io/projected/ecd04dfe-4f45-4872-995a-060a166d25f8-kube-api-access-k6nbs\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.130045 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.130031    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db7b07b-4df5-4a3e-b273-2f5698b8d317-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:30:59.133828 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.133794    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:59.231242 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.231204    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.231242 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.231251    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecd04dfe-4f45-4872-995a-060a166d25f8-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.231602 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.231325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6nbs\" (UniqueName: \"kubernetes.io/projected/ecd04dfe-4f45-4872-995a-060a166d25f8-kube-api-access-k6nbs\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.231602 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.231363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.231602 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.231431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-dshm\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.231602 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.231476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-home\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.231829 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.231783    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.231888 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.231826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.231888 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.231866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-home\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.234123 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.234099    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-dshm\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.234361 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.234342    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecd04dfe-4f45-4872-995a-060a166d25f8-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.241349 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.241320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6nbs\" (UniqueName: \"kubernetes.io/projected/ecd04dfe-4f45-4872-995a-060a166d25f8-kube-api-access-k6nbs\") pod \"roll-v2-kserve-d4dbf9d67-9xhq7\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.313255 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.313215    2582 generic.go:356] "Generic (PLEG): container finished" podID="8db7b07b-4df5-4a3e-b273-2f5698b8d317" containerID="b04ce0051de9e0e399fd4dd6bac74c1ce15561b9e6f116eb34f1ed5c5c0c9772" exitCode=0
Sep 16 19:30:59.313255 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.313260    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd" event={"ID":"8db7b07b-4df5-4a3e-b273-2f5698b8d317","Type":"ContainerDied","Data":"b04ce0051de9e0e399fd4dd6bac74c1ce15561b9e6f116eb34f1ed5c5c0c9772"}
Sep 16 19:30:59.313543 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.313271    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"
Sep 16 19:30:59.313543 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.313313    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd" event={"ID":"8db7b07b-4df5-4a3e-b273-2f5698b8d317","Type":"ContainerDied","Data":"a978ffcabe755163132fdcca491732ef14248eb53ce025b0012cf4a9f063e453"}
Sep 16 19:30:59.313543 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.313339    2582 scope.go:117] "RemoveContainer" containerID="b04ce0051de9e0e399fd4dd6bac74c1ce15561b9e6f116eb34f1ed5c5c0c9772"
Sep 16 19:30:59.323189 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.323170    2582 scope.go:117] "RemoveContainer" containerID="b04ce0051de9e0e399fd4dd6bac74c1ce15561b9e6f116eb34f1ed5c5c0c9772"
Sep 16 19:30:59.323520 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:30:59.323500    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04ce0051de9e0e399fd4dd6bac74c1ce15561b9e6f116eb34f1ed5c5c0c9772\": container with ID starting with b04ce0051de9e0e399fd4dd6bac74c1ce15561b9e6f116eb34f1ed5c5c0c9772 not found: ID does not exist" containerID="b04ce0051de9e0e399fd4dd6bac74c1ce15561b9e6f116eb34f1ed5c5c0c9772"
Sep 16 19:30:59.323611 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.323541    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04ce0051de9e0e399fd4dd6bac74c1ce15561b9e6f116eb34f1ed5c5c0c9772"} err="failed to get container status \"b04ce0051de9e0e399fd4dd6bac74c1ce15561b9e6f116eb34f1ed5c5c0c9772\": rpc error: code = NotFound desc = could not find container \"b04ce0051de9e0e399fd4dd6bac74c1ce15561b9e6f116eb34f1ed5c5c0c9772\": container with ID starting with b04ce0051de9e0e399fd4dd6bac74c1ce15561b9e6f116eb34f1ed5c5c0c9772 not found: ID does not exist"
Sep 16 19:30:59.337851 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.337813    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"]
Sep 16 19:30:59.341809 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.341785    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd"]
Sep 16 19:30:59.413494 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.413399    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:30:59.427345 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.427312    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:59.441647 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.441613    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:30:59.704163 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.704090    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"]
Sep 16 19:30:59.708042 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:30:59.708001    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd04dfe_4f45_4872_995a_060a166d25f8.slice/crio-1cb130ef91927f3d2748dcf0d48db5ff92ba0567841d425ad5f54ab19e4ec7f3 WatchSource:0}: Error finding container 1cb130ef91927f3d2748dcf0d48db5ff92ba0567841d425ad5f54ab19e4ec7f3: Status 404 returned error can't find the container with id 1cb130ef91927f3d2748dcf0d48db5ff92ba0567841d425ad5f54ab19e4ec7f3
Sep 16 19:30:59.835863 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:30:59.835804    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-wz8cd" podUID="8db7b07b-4df5-4a3e-b273-2f5698b8d317" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded"
Sep 16 19:31:00.321374 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:00.321323    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7" event={"ID":"ecd04dfe-4f45-4872-995a-060a166d25f8","Type":"ContainerStarted","Data":"3a1a044f17d24825588f82780a671c304dcb1ccd359156b9bc9573c23e03578e"}
Sep 16 19:31:00.321649 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:00.321400    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7" event={"ID":"ecd04dfe-4f45-4872-995a-060a166d25f8","Type":"ContainerStarted","Data":"1cb130ef91927f3d2748dcf0d48db5ff92ba0567841d425ad5f54ab19e4ec7f3"}
Sep 16 19:31:00.332748 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:00.332709    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:31:00.342171 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:00.342110    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7" podStartSLOduration=1.342093061 podStartE2EDuration="1.342093061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:00.339916496 +0000 UTC m=+2718.170525044" watchObservedRunningTime="2026-09-16 19:31:00.342093061 +0000 UTC m=+2718.172701622"
Sep 16 19:31:00.413979 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:00.413938    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:31:00.704169 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:00.704052    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8db7b07b-4df5-4a3e-b273-2f5698b8d317" path="/var/lib/kubelet/pods/8db7b07b-4df5-4a3e-b273-2f5698b8d317/volumes"
Sep 16 19:31:01.682562 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:31:01.682502    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a0ab4_1ae1_4b97_8bb0_7ce87a1c63e6.slice/crio-ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:31:06.559366 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:31:06.559310    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a0ab4_1ae1_4b97_8bb0_7ce87a1c63e6.slice/crio-ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:31:09.414233 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:09.414160    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:31:09.428591 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:09.428534    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:31:10.384197 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:10.384161    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:31:11.732940 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:31:11.732884    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a0ab4_1ae1_4b97_8bb0_7ce87a1c63e6.slice/crio-ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:31:12.519962 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:12.519916    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"]
Sep 16 19:31:12.523953 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:12.523917    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:12.617565 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:12.617527    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:13.505256 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:13.505204    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p" podUID="8edba542-89ca-4b0c-a28d-4fafeea97647" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:21.560145 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:31:21.560084    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a0ab4_1ae1_4b97_8bb0_7ce87a1c63e6.slice/crio-ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:31:21.743498 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:31:21.743448    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a0ab4_1ae1_4b97_8bb0_7ce87a1c63e6.slice/crio-ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:31:23.451040 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:31:23.450997    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a0ab4_1ae1_4b97_8bb0_7ce87a1c63e6.slice/crio-ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:31:23.451773 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:31:23.451744    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a0ab4_1ae1_4b97_8bb0_7ce87a1c63e6.slice/crio-ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:31:23.505051 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:23.505005    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p" podUID="8edba542-89ca-4b0c-a28d-4fafeea97647" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:25.746419 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:25.746384    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:27.542863 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:27.542808    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p" podUID="8edba542-89ca-4b0c-a28d-4fafeea97647" containerName="main" containerID="cri-o://ec2a64afaa39b4a81205fa6bb9801461921f8c73fdf1b281f988e22e4d33e0b2" gracePeriod=45
Sep 16 19:31:31.362179 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:31.362126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:31.435392 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:31.435350    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:31.790860 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:31:31.790814    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a0ab4_1ae1_4b97_8bb0_7ce87a1c63e6.slice/crio-ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:31:33.505351 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:33.505305    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p" podUID="8edba542-89ca-4b0c-a28d-4fafeea97647" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:33.505857 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:33.505437    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"
Sep 16 19:31:34.901044 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:34.900997    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:34.967543 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:34.967505    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:36.579119 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:31:36.579072    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a0ab4_1ae1_4b97_8bb0_7ce87a1c63e6.slice/crio-ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:31:41.836786 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:31:41.836746    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7a0ab4_1ae1_4b97_8bb0_7ce87a1c63e6.slice/crio-ea676790bd56890e341268be5e431474ba92f85d20cde414abab0c959cec03ad.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:31:43.504848 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:43.504801    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p" podUID="8edba542-89ca-4b0c-a28d-4fafeea97647" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:53.505179 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:53.505136    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p" podUID="8edba542-89ca-4b0c-a28d-4fafeea97647" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:57.797554 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:57.797521    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"
Sep 16 19:31:57.899275 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:57.899158    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8edba542-89ca-4b0c-a28d-4fafeea97647-tls-certs\") pod \"8edba542-89ca-4b0c-a28d-4fafeea97647\" (UID: \"8edba542-89ca-4b0c-a28d-4fafeea97647\") "
Sep 16 19:31:57.899275 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:57.899268    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqz6f\" (UniqueName: \"kubernetes.io/projected/8edba542-89ca-4b0c-a28d-4fafeea97647-kube-api-access-zqz6f\") pod \"8edba542-89ca-4b0c-a28d-4fafeea97647\" (UID: \"8edba542-89ca-4b0c-a28d-4fafeea97647\") "
Sep 16 19:31:57.901882 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:57.901838    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8edba542-89ca-4b0c-a28d-4fafeea97647-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8edba542-89ca-4b0c-a28d-4fafeea97647" (UID: "8edba542-89ca-4b0c-a28d-4fafeea97647"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:57.902033 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:57.901962    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8edba542-89ca-4b0c-a28d-4fafeea97647-kube-api-access-zqz6f" (OuterVolumeSpecName: "kube-api-access-zqz6f") pod "8edba542-89ca-4b0c-a28d-4fafeea97647" (UID: "8edba542-89ca-4b0c-a28d-4fafeea97647"). InnerVolumeSpecName "kube-api-access-zqz6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:58.000910 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:58.000849    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8edba542-89ca-4b0c-a28d-4fafeea97647-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:31:58.000910 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:58.000898    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqz6f\" (UniqueName: \"kubernetes.io/projected/8edba542-89ca-4b0c-a28d-4fafeea97647-kube-api-access-zqz6f\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:31:58.626851 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:58.626810    2582 generic.go:356] "Generic (PLEG): container finished" podID="8edba542-89ca-4b0c-a28d-4fafeea97647" containerID="ec2a64afaa39b4a81205fa6bb9801461921f8c73fdf1b281f988e22e4d33e0b2" exitCode=0
Sep 16 19:31:58.626851 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:58.626858    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"
Sep 16 19:31:58.627113 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:58.626891    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p" event={"ID":"8edba542-89ca-4b0c-a28d-4fafeea97647","Type":"ContainerDied","Data":"ec2a64afaa39b4a81205fa6bb9801461921f8c73fdf1b281f988e22e4d33e0b2"}
Sep 16 19:31:58.627113 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:58.626951    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p" event={"ID":"8edba542-89ca-4b0c-a28d-4fafeea97647","Type":"ContainerDied","Data":"193ed0e44f21ce7d839b2a2a3df70b15466695777dfa2afa35cd1aaec0aefc2e"}
Sep 16 19:31:58.627113 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:58.626978    2582 scope.go:117] "RemoveContainer" containerID="ec2a64afaa39b4a81205fa6bb9801461921f8c73fdf1b281f988e22e4d33e0b2"
Sep 16 19:31:58.637412 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:58.637389    2582 scope.go:117] "RemoveContainer" containerID="ec2a64afaa39b4a81205fa6bb9801461921f8c73fdf1b281f988e22e4d33e0b2"
Sep 16 19:31:58.637772 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:31:58.637739    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2a64afaa39b4a81205fa6bb9801461921f8c73fdf1b281f988e22e4d33e0b2\": container with ID starting with ec2a64afaa39b4a81205fa6bb9801461921f8c73fdf1b281f988e22e4d33e0b2 not found: ID does not exist" containerID="ec2a64afaa39b4a81205fa6bb9801461921f8c73fdf1b281f988e22e4d33e0b2"
Sep 16 19:31:58.637877 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:58.637778    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2a64afaa39b4a81205fa6bb9801461921f8c73fdf1b281f988e22e4d33e0b2"} err="failed to get container status \"ec2a64afaa39b4a81205fa6bb9801461921f8c73fdf1b281f988e22e4d33e0b2\": rpc error: code = NotFound desc = could not find container \"ec2a64afaa39b4a81205fa6bb9801461921f8c73fdf1b281f988e22e4d33e0b2\": container with ID starting with ec2a64afaa39b4a81205fa6bb9801461921f8c73fdf1b281f988e22e4d33e0b2 not found: ID does not exist"
Sep 16 19:31:58.657933 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:58.657898    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"]
Sep 16 19:31:58.659724 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:58.659698    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schevr48p"]
Sep 16 19:31:58.704586 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:31:58.704531    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8edba542-89ca-4b0c-a28d-4fafeea97647" path="/var/lib/kubelet/pods/8edba542-89ca-4b0c-a28d-4fafeea97647/volumes"
Sep 16 19:32:00.598951 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:00.598912    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:00.695605 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:00.695540    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:02.153616 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:02.153564    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:02.260496 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:02.260468    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:07.003800 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:07.003710    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"]
Sep 16 19:32:07.009477 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:07.009439    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"]
Sep 16 19:32:07.101135 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:07.101104    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:07.201319 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:07.201281    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:12.870217 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:12.870181    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:12.972265 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:12.972217    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:22.024509 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.024440    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm" podUID="ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8" containerName="main" containerID="cri-o://7ebdbbb3a10f1127d27bf3901d5041c8a365a4fe4d23c2587460444f7818da0c" gracePeriod=45
Sep 16 19:32:22.029792 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.029373    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7" podUID="ecd04dfe-4f45-4872-995a-060a166d25f8" containerName="main" containerID="cri-o://3a1a044f17d24825588f82780a671c304dcb1ccd359156b9bc9573c23e03578e" gracePeriod=45
Sep 16 19:32:22.287032 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.286946    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:32:22.307904 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.307868    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:32:22.319089 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319058    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecd04dfe-4f45-4872-995a-060a166d25f8-tls-certs\") pod \"ecd04dfe-4f45-4872-995a-060a166d25f8\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") "
Sep 16 19:32:22.319313 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319147    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-dshm\") pod \"ecd04dfe-4f45-4872-995a-060a166d25f8\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") "
Sep 16 19:32:22.319313 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319217    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-tmp-dir\") pod \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") "
Sep 16 19:32:22.319313 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319252    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-home\") pod \"ecd04dfe-4f45-4872-995a-060a166d25f8\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") "
Sep 16 19:32:22.319313 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319274    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-model-cache\") pod \"ecd04dfe-4f45-4872-995a-060a166d25f8\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") "
Sep 16 19:32:22.319544 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319322    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-model-cache\") pod \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") "
Sep 16 19:32:22.319544 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319347    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-home\") pod \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") "
Sep 16 19:32:22.319544 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319377    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dljtk\" (UniqueName: \"kubernetes.io/projected/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-kube-api-access-dljtk\") pod \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") "
Sep 16 19:32:22.319544 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319431    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-dshm\") pod \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") "
Sep 16 19:32:22.319544 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319459    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-tls-certs\") pod \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\" (UID: \"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8\") "
Sep 16 19:32:22.319544 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319484    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-tmp-dir\") pod \"ecd04dfe-4f45-4872-995a-060a166d25f8\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") "
Sep 16 19:32:22.319544 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319519    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8" (UID: "ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:22.319544 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319545    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6nbs\" (UniqueName: \"kubernetes.io/projected/ecd04dfe-4f45-4872-995a-060a166d25f8-kube-api-access-k6nbs\") pod \"ecd04dfe-4f45-4872-995a-060a166d25f8\" (UID: \"ecd04dfe-4f45-4872-995a-060a166d25f8\") "
Sep 16 19:32:22.320136 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319678    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8" (UID: "ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:22.320136 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319939    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:32:22.320136 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.319962    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:32:22.320918 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.320355    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-home" (OuterVolumeSpecName: "home") pod "ecd04dfe-4f45-4872-995a-060a166d25f8" (UID: "ecd04dfe-4f45-4872-995a-060a166d25f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:22.320918 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.320627    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-home" (OuterVolumeSpecName: "home") pod "ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8" (UID: "ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:22.320918 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.320878    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ecd04dfe-4f45-4872-995a-060a166d25f8" (UID: "ecd04dfe-4f45-4872-995a-060a166d25f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:22.321276 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.321226    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "ecd04dfe-4f45-4872-995a-060a166d25f8" (UID: "ecd04dfe-4f45-4872-995a-060a166d25f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:22.322317 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.322290    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-dshm" (OuterVolumeSpecName: "dshm") pod "ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8" (UID: "ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:22.322688 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.322615    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd04dfe-4f45-4872-995a-060a166d25f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ecd04dfe-4f45-4872-995a-060a166d25f8" (UID: "ecd04dfe-4f45-4872-995a-060a166d25f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:22.322688 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.322619    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd04dfe-4f45-4872-995a-060a166d25f8-kube-api-access-k6nbs" (OuterVolumeSpecName: "kube-api-access-k6nbs") pod "ecd04dfe-4f45-4872-995a-060a166d25f8" (UID: "ecd04dfe-4f45-4872-995a-060a166d25f8"). InnerVolumeSpecName "kube-api-access-k6nbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:22.322848 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.322700    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-dshm" (OuterVolumeSpecName: "dshm") pod "ecd04dfe-4f45-4872-995a-060a166d25f8" (UID: "ecd04dfe-4f45-4872-995a-060a166d25f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:22.323050 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.323011    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8" (UID: "ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:22.324106 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.324070    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-kube-api-access-dljtk" (OuterVolumeSpecName: "kube-api-access-dljtk") pod "ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8" (UID: "ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8"). InnerVolumeSpecName "kube-api-access-dljtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:22.420812 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.420768    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:32:22.420812 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.420805    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:32:22.420812 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.420816    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:32:22.420812 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.420825    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6nbs\" (UniqueName: \"kubernetes.io/projected/ecd04dfe-4f45-4872-995a-060a166d25f8-kube-api-access-k6nbs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:32:22.421103 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.420835    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecd04dfe-4f45-4872-995a-060a166d25f8-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:32:22.421103 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.420853    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:32:22.421103 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.420861    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:32:22.421103 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.420869    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecd04dfe-4f45-4872-995a-060a166d25f8-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:32:22.421103 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.420876    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:32:22.421103 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.420883    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dljtk\" (UniqueName: \"kubernetes.io/projected/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8-kube-api-access-dljtk\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:32:22.744053 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.744018    2582 generic.go:356] "Generic (PLEG): container finished" podID="ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8" containerID="7ebdbbb3a10f1127d27bf3901d5041c8a365a4fe4d23c2587460444f7818da0c" exitCode=0
Sep 16 19:32:22.744196 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.744066    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"
Sep 16 19:32:22.744196 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.744107    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm" event={"ID":"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8","Type":"ContainerDied","Data":"7ebdbbb3a10f1127d27bf3901d5041c8a365a4fe4d23c2587460444f7818da0c"}
Sep 16 19:32:22.744196 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.744162    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm" event={"ID":"ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8","Type":"ContainerDied","Data":"4bf7c408d4b54516a131f2e3872810fc54c019e5adc427e6e1cfdb72801eae86"}
Sep 16 19:32:22.744196 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.744193    2582 scope.go:117] "RemoveContainer" containerID="7ebdbbb3a10f1127d27bf3901d5041c8a365a4fe4d23c2587460444f7818da0c"
Sep 16 19:32:22.752100 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.752067    2582 generic.go:356] "Generic (PLEG): container finished" podID="ecd04dfe-4f45-4872-995a-060a166d25f8" containerID="3a1a044f17d24825588f82780a671c304dcb1ccd359156b9bc9573c23e03578e" exitCode=0
Sep 16 19:32:22.752309 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.752175    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7" event={"ID":"ecd04dfe-4f45-4872-995a-060a166d25f8","Type":"ContainerDied","Data":"3a1a044f17d24825588f82780a671c304dcb1ccd359156b9bc9573c23e03578e"}
Sep 16 19:32:22.752634 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.752208    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"
Sep 16 19:32:22.752755 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.752702    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7" event={"ID":"ecd04dfe-4f45-4872-995a-060a166d25f8","Type":"ContainerDied","Data":"1cb130ef91927f3d2748dcf0d48db5ff92ba0567841d425ad5f54ab19e4ec7f3"}
Sep 16 19:32:22.760124 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.760105    2582 scope.go:117] "RemoveContainer" containerID="7ebdbbb3a10f1127d27bf3901d5041c8a365a4fe4d23c2587460444f7818da0c"
Sep 16 19:32:22.760471 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:32:22.760453    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ebdbbb3a10f1127d27bf3901d5041c8a365a4fe4d23c2587460444f7818da0c\": container with ID starting with 7ebdbbb3a10f1127d27bf3901d5041c8a365a4fe4d23c2587460444f7818da0c not found: ID does not exist" containerID="7ebdbbb3a10f1127d27bf3901d5041c8a365a4fe4d23c2587460444f7818da0c"
Sep 16 19:32:22.760534 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.760479    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ebdbbb3a10f1127d27bf3901d5041c8a365a4fe4d23c2587460444f7818da0c"} err="failed to get container status \"7ebdbbb3a10f1127d27bf3901d5041c8a365a4fe4d23c2587460444f7818da0c\": rpc error: code = NotFound desc = could not find container \"7ebdbbb3a10f1127d27bf3901d5041c8a365a4fe4d23c2587460444f7818da0c\": container with ID starting with 7ebdbbb3a10f1127d27bf3901d5041c8a365a4fe4d23c2587460444f7818da0c not found: ID does not exist"
Sep 16 19:32:22.760534 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.760496    2582 scope.go:117] "RemoveContainer" containerID="3a1a044f17d24825588f82780a671c304dcb1ccd359156b9bc9573c23e03578e"
Sep 16 19:32:22.768917 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.768896    2582 scope.go:117] "RemoveContainer" containerID="3a1a044f17d24825588f82780a671c304dcb1ccd359156b9bc9573c23e03578e"
Sep 16 19:32:22.769231 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:32:22.769206    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1a044f17d24825588f82780a671c304dcb1ccd359156b9bc9573c23e03578e\": container with ID starting with 3a1a044f17d24825588f82780a671c304dcb1ccd359156b9bc9573c23e03578e not found: ID does not exist" containerID="3a1a044f17d24825588f82780a671c304dcb1ccd359156b9bc9573c23e03578e"
Sep 16 19:32:22.769322 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.769237    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1a044f17d24825588f82780a671c304dcb1ccd359156b9bc9573c23e03578e"} err="failed to get container status \"3a1a044f17d24825588f82780a671c304dcb1ccd359156b9bc9573c23e03578e\": rpc error: code = NotFound desc = could not find container \"3a1a044f17d24825588f82780a671c304dcb1ccd359156b9bc9573c23e03578e\": container with ID starting with 3a1a044f17d24825588f82780a671c304dcb1ccd359156b9bc9573c23e03578e not found: ID does not exist"
Sep 16 19:32:22.769632 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.769610    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"]
Sep 16 19:32:22.773779 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.773757    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-9xhq7"]
Sep 16 19:32:22.786316 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.786292    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"]
Sep 16 19:32:22.787726 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:22.787707    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-zcjjm"]
Sep 16 19:32:24.703513 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.703477    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecd04dfe-4f45-4872-995a-060a166d25f8" path="/var/lib/kubelet/pods/ecd04dfe-4f45-4872-995a-060a166d25f8/volumes"
Sep 16 19:32:24.704018 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.704000    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8" path="/var/lib/kubelet/pods/ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8/volumes"
Sep 16 19:32:24.764758 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.764722    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:24.933772 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.933735    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"]
Sep 16 19:32:24.934117 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.934101    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8" containerName="main"
Sep 16 19:32:24.934117 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.934119    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8" containerName="main"
Sep 16 19:32:24.934210 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.934130    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecd04dfe-4f45-4872-995a-060a166d25f8" containerName="main"
Sep 16 19:32:24.934210 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.934135    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecd04dfe-4f45-4872-995a-060a166d25f8" containerName="main"
Sep 16 19:32:24.934210 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.934156    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8edba542-89ca-4b0c-a28d-4fafeea97647" containerName="main"
Sep 16 19:32:24.934210 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.934162    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8edba542-89ca-4b0c-a28d-4fafeea97647" containerName="main"
Sep 16 19:32:24.934333 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.934213    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8edba542-89ca-4b0c-a28d-4fafeea97647" containerName="main"
Sep 16 19:32:24.934333 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.934223    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffb62a1a-00a2-44ef-9a10-b0a53c2a13a8" containerName="main"
Sep 16 19:32:24.934333 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.934230    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecd04dfe-4f45-4872-995a-060a166d25f8" containerName="main"
Sep 16 19:32:24.938870 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.938852    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"
Sep 16 19:32:24.943521 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.943489    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 19:32:24.943728 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.943502    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 19:32:24.943728 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.943558    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-mwnvw\""
Sep 16 19:32:24.943728 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.943524    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 19:32:24.943728 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.943502    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-dzg4f\""
Sep 16 19:32:24.951055 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.951018    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"]
Sep 16 19:32:24.970750 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:24.970676    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:25.042218 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:25.042160    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f3b533-f0b2-4c99-93d1-eb47070a29e0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2\" (UID: \"f7f3b533-f0b2-4c99-93d1-eb47070a29e0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"
Sep 16 19:32:25.042401 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:25.042249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h97xr\" (UniqueName: \"kubernetes.io/projected/f7f3b533-f0b2-4c99-93d1-eb47070a29e0-kube-api-access-h97xr\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2\" (UID: \"f7f3b533-f0b2-4c99-93d1-eb47070a29e0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"
Sep 16 19:32:25.142999 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:25.142959    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f3b533-f0b2-4c99-93d1-eb47070a29e0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2\" (UID: \"f7f3b533-f0b2-4c99-93d1-eb47070a29e0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"
Sep 16 19:32:25.143188 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:25.143028    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h97xr\" (UniqueName: \"kubernetes.io/projected/f7f3b533-f0b2-4c99-93d1-eb47070a29e0-kube-api-access-h97xr\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2\" (UID: \"f7f3b533-f0b2-4c99-93d1-eb47070a29e0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"
Sep 16 19:32:25.145799 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:25.145764    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f3b533-f0b2-4c99-93d1-eb47070a29e0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2\" (UID: \"f7f3b533-f0b2-4c99-93d1-eb47070a29e0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"
Sep 16 19:32:25.152301 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:25.152274    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h97xr\" (UniqueName: \"kubernetes.io/projected/f7f3b533-f0b2-4c99-93d1-eb47070a29e0-kube-api-access-h97xr\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2\" (UID: \"f7f3b533-f0b2-4c99-93d1-eb47070a29e0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"
Sep 16 19:32:25.252761 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:25.252658    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"
Sep 16 19:32:25.528589 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:25.528527    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"]
Sep 16 19:32:25.531171 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:32:25.531141    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f3b533_f0b2_4c99_93d1_eb47070a29e0.slice/crio-197edf3340018147fbb77c1815803cd64ef0029d7d98b1e97db50d3bda340393 WatchSource:0}: Error finding container 197edf3340018147fbb77c1815803cd64ef0029d7d98b1e97db50d3bda340393: Status 404 returned error can't find the container with id 197edf3340018147fbb77c1815803cd64ef0029d7d98b1e97db50d3bda340393
Sep 16 19:32:25.774322 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:25.774282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2" event={"ID":"f7f3b533-f0b2-4c99-93d1-eb47070a29e0","Type":"ContainerStarted","Data":"570efe446205c4dc08553b7badc7c6f0786081293b9d71e3f87609cba60cc929"}
Sep 16 19:32:25.774808 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:25.774349    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2" event={"ID":"f7f3b533-f0b2-4c99-93d1-eb47070a29e0","Type":"ContainerStarted","Data":"197edf3340018147fbb77c1815803cd64ef0029d7d98b1e97db50d3bda340393"}
Sep 16 19:32:25.774808 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:25.774399    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"
Sep 16 19:32:25.800417 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:25.800301    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2" podStartSLOduration=1.800281808 podStartE2EDuration="1.800281808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:25.798334155 +0000 UTC m=+2803.628942703" watchObservedRunningTime="2026-09-16 19:32:25.800281808 +0000 UTC m=+2803.630890353"
Sep 16 19:32:33.919317 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:33.919270    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:38.925040 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:38.925006    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:40.327807 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:40.327765    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:41.105726 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:41.105694    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:44.144914 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:44.144874    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:47.209324 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.209285    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"]
Sep 16 19:32:47.214058 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.214034    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.218337 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.218314    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 16 19:32:47.219840 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.219816    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 16 19:32:47.219840 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.219831    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-x8k6b\""
Sep 16 19:32:47.220019 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.219938    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Sep 16 19:32:47.229151 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.229129    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"]
Sep 16 19:32:47.256305 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.256263    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:47.344059 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.344022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-home\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.344059 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.344061    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl6dq\" (UniqueName: \"kubernetes.io/projected/bb30978c-72b5-4f46-a040-58ad9dcce859-kube-api-access-sl6dq\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.344292 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.344095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-model-cache\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.344292 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.344205    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb30978c-72b5-4f46-a040-58ad9dcce859-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.344292 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.344272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-dshm\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.344398 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.344316    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.445409 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.445355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-dshm\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.445644 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.445447    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.445644 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.445477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-home\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.445644 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.445513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl6dq\" (UniqueName: \"kubernetes.io/projected/bb30978c-72b5-4f46-a040-58ad9dcce859-kube-api-access-sl6dq\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.445644 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.445549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-model-cache\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.445820 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.445646    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb30978c-72b5-4f46-a040-58ad9dcce859-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.445991 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.445968    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.446068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.446011    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-home\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.446068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.446036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-model-cache\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.448245 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.448212    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-dshm\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.448367 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.448352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb30978c-72b5-4f46-a040-58ad9dcce859-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.454279 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.454247    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl6dq\" (UniqueName: \"kubernetes.io/projected/bb30978c-72b5-4f46-a040-58ad9dcce859-kube-api-access-sl6dq\") pod \"owner-v1-kserve-75579ff5bc-whqkl\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.524703 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.524668    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:47.808530 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.808492    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"]
Sep 16 19:32:47.810062 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:32:47.810025    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb30978c_72b5_4f46_a040_58ad9dcce859.slice/crio-1eb759f407a920cc70cc8a31b7cf2511f3bd3ccd0563a917f738bc868ee37bc4 WatchSource:0}: Error finding container 1eb759f407a920cc70cc8a31b7cf2511f3bd3ccd0563a917f738bc868ee37bc4: Status 404 returned error can't find the container with id 1eb759f407a920cc70cc8a31b7cf2511f3bd3ccd0563a917f738bc868ee37bc4
Sep 16 19:32:47.884935 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:47.884876    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl" event={"ID":"bb30978c-72b5-4f46-a040-58ad9dcce859","Type":"ContainerStarted","Data":"1eb759f407a920cc70cc8a31b7cf2511f3bd3ccd0563a917f738bc868ee37bc4"}
Sep 16 19:32:48.891491 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:48.891448    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl" event={"ID":"bb30978c-72b5-4f46-a040-58ad9dcce859","Type":"ContainerStarted","Data":"955394a58e9b500a97b572348b15e30ef9060c93a701096f53ad352837502596"}
Sep 16 19:32:48.910366 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:48.910310    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl" podStartSLOduration=1.910289352 podStartE2EDuration="1.910289352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:48.909746008 +0000 UTC m=+2826.740354554" watchObservedRunningTime="2026-09-16 19:32:48.910289352 +0000 UTC m=+2826.740897897"
Sep 16 19:32:49.525529 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:49.525475    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:49.639673 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:49.639638    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:49.739977 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:49.739941    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:55.707073 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.707036    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"]
Sep 16 19:32:55.710508 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.710486    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.713992 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.713966    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Sep 16 19:32:55.721753 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.721724    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"]
Sep 16 19:32:55.751440 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.751401    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:55.824291 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.824248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.824291 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.824288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.824565 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.824308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-dshm\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.824565 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.824327    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-model-cache\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.824565 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.824348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-home\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.824565 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.824378    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqmrt\" (UniqueName: \"kubernetes.io/projected/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-kube-api-access-kqmrt\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.925528 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.925490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.925528 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.925526    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.925844 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.925546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-dshm\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.925844 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.925662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-model-cache\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.925844 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.925718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-home\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.925844 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.925777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmrt\" (UniqueName: \"kubernetes.io/projected/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-kube-api-access-kqmrt\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.926051 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.925998    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.926117 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.926094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-model-cache\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.926169 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.926150    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-home\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.928276 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.928252    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-dshm\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.928544 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.928526    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:55.934799 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:55.934772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqmrt\" (UniqueName: \"kubernetes.io/projected/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-kube-api-access-kqmrt\") pod \"owner-v2-kserve-5cf8959f95-5zwgm\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:56.022846 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:56.022803    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:56.325431 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:56.325343    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"]
Sep 16 19:32:56.329401 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:32:56.329368    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02f7f74_85ef_47c7_b5cf_e5d321ebdd86.slice/crio-06b00855549805f5fbc313ef17a0a572d690c586ce661bf83cc7925f25d6fe9b WatchSource:0}: Error finding container 06b00855549805f5fbc313ef17a0a572d690c586ce661bf83cc7925f25d6fe9b: Status 404 returned error can't find the container with id 06b00855549805f5fbc313ef17a0a572d690c586ce661bf83cc7925f25d6fe9b
Sep 16 19:32:56.781501 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:56.781469    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"
Sep 16 19:32:56.930749 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:56.930705    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm" event={"ID":"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86","Type":"ContainerStarted","Data":"8eeb54ec20b9738eaf209a567fa30408b5c4c323828b1a64c277de5870f0ed86"}
Sep 16 19:32:56.930946 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:56.930772    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm" event={"ID":"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86","Type":"ContainerStarted","Data":"06b00855549805f5fbc313ef17a0a572d690c586ce661bf83cc7925f25d6fe9b"}
Sep 16 19:32:56.950751 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:56.950697    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm" podStartSLOduration=1.950678423 podStartE2EDuration="1.950678423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:56.949996616 +0000 UTC m=+2834.780605165" watchObservedRunningTime="2026-09-16 19:32:56.950678423 +0000 UTC m=+2834.781286971"
Sep 16 19:32:57.022977 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:57.022935    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:32:57.525106 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:57.525070    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:57.537727 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:57.537699    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:32:57.947845 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:32:57.947751    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:33:06.023926 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:33:06.023879    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:33:06.036561 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:33:06.036531    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:33:06.988554 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:33:06.988521    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:33:11.437843 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:33:11.437795    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:11.495024 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:33:11.494984    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:48.379736 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:33:48.379655    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"]
Sep 16 19:33:48.403775 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:33:48.403745    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:03.399551 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.399505    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl" podUID="bb30978c-72b5-4f46-a040-58ad9dcce859" containerName="main" containerID="cri-o://955394a58e9b500a97b572348b15e30ef9060c93a701096f53ad352837502596" gracePeriod=45
Sep 16 19:34:03.662730 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.662654    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:34:03.734288 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.734238    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-tmp-dir\") pod \"bb30978c-72b5-4f46-a040-58ad9dcce859\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") "
Sep 16 19:34:03.734288 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.734293    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl6dq\" (UniqueName: \"kubernetes.io/projected/bb30978c-72b5-4f46-a040-58ad9dcce859-kube-api-access-sl6dq\") pod \"bb30978c-72b5-4f46-a040-58ad9dcce859\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") "
Sep 16 19:34:03.734603 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.734333    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-dshm\") pod \"bb30978c-72b5-4f46-a040-58ad9dcce859\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") "
Sep 16 19:34:03.734603 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.734360    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb30978c-72b5-4f46-a040-58ad9dcce859-tls-certs\") pod \"bb30978c-72b5-4f46-a040-58ad9dcce859\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") "
Sep 16 19:34:03.734603 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.734387    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-home\") pod \"bb30978c-72b5-4f46-a040-58ad9dcce859\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") "
Sep 16 19:34:03.734603 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.734439    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-model-cache\") pod \"bb30978c-72b5-4f46-a040-58ad9dcce859\" (UID: \"bb30978c-72b5-4f46-a040-58ad9dcce859\") "
Sep 16 19:34:03.734830 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.734640    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb30978c-72b5-4f46-a040-58ad9dcce859" (UID: "bb30978c-72b5-4f46-a040-58ad9dcce859"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:03.734830 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.734750    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-home" (OuterVolumeSpecName: "home") pod "bb30978c-72b5-4f46-a040-58ad9dcce859" (UID: "bb30978c-72b5-4f46-a040-58ad9dcce859"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:03.734925 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.734880    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.734925 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.734901    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.735047 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.734978    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb30978c-72b5-4f46-a040-58ad9dcce859" (UID: "bb30978c-72b5-4f46-a040-58ad9dcce859"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:03.737015 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.736991    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-dshm" (OuterVolumeSpecName: "dshm") pod "bb30978c-72b5-4f46-a040-58ad9dcce859" (UID: "bb30978c-72b5-4f46-a040-58ad9dcce859"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:03.737143 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.737021    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb30978c-72b5-4f46-a040-58ad9dcce859-kube-api-access-sl6dq" (OuterVolumeSpecName: "kube-api-access-sl6dq") pod "bb30978c-72b5-4f46-a040-58ad9dcce859" (UID: "bb30978c-72b5-4f46-a040-58ad9dcce859"). InnerVolumeSpecName "kube-api-access-sl6dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:03.737143 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.737046    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb30978c-72b5-4f46-a040-58ad9dcce859-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb30978c-72b5-4f46-a040-58ad9dcce859" (UID: "bb30978c-72b5-4f46-a040-58ad9dcce859"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:03.835831 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.835787    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl6dq\" (UniqueName: \"kubernetes.io/projected/bb30978c-72b5-4f46-a040-58ad9dcce859-kube-api-access-sl6dq\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.835831 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.835824    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.835831 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.835836    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb30978c-72b5-4f46-a040-58ad9dcce859-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.836107 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:03.835848    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb30978c-72b5-4f46-a040-58ad9dcce859-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:34:04.256538 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:04.256497    2582 generic.go:356] "Generic (PLEG): container finished" podID="bb30978c-72b5-4f46-a040-58ad9dcce859" containerID="955394a58e9b500a97b572348b15e30ef9060c93a701096f53ad352837502596" exitCode=0
Sep 16 19:34:04.256755 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:04.256561    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"
Sep 16 19:34:04.256755 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:04.256611    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl" event={"ID":"bb30978c-72b5-4f46-a040-58ad9dcce859","Type":"ContainerDied","Data":"955394a58e9b500a97b572348b15e30ef9060c93a701096f53ad352837502596"}
Sep 16 19:34:04.256755 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:04.256704    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl" event={"ID":"bb30978c-72b5-4f46-a040-58ad9dcce859","Type":"ContainerDied","Data":"1eb759f407a920cc70cc8a31b7cf2511f3bd3ccd0563a917f738bc868ee37bc4"}
Sep 16 19:34:04.256755 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:04.256733    2582 scope.go:117] "RemoveContainer" containerID="955394a58e9b500a97b572348b15e30ef9060c93a701096f53ad352837502596"
Sep 16 19:34:04.266443 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:04.266421    2582 scope.go:117] "RemoveContainer" containerID="955394a58e9b500a97b572348b15e30ef9060c93a701096f53ad352837502596"
Sep 16 19:34:04.266802 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:34:04.266783    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"955394a58e9b500a97b572348b15e30ef9060c93a701096f53ad352837502596\": container with ID starting with 955394a58e9b500a97b572348b15e30ef9060c93a701096f53ad352837502596 not found: ID does not exist" containerID="955394a58e9b500a97b572348b15e30ef9060c93a701096f53ad352837502596"
Sep 16 19:34:04.266893 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:04.266812    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"955394a58e9b500a97b572348b15e30ef9060c93a701096f53ad352837502596"} err="failed to get container status \"955394a58e9b500a97b572348b15e30ef9060c93a701096f53ad352837502596\": rpc error: code = NotFound desc = could not find container \"955394a58e9b500a97b572348b15e30ef9060c93a701096f53ad352837502596\": container with ID starting with 955394a58e9b500a97b572348b15e30ef9060c93a701096f53ad352837502596 not found: ID does not exist"
Sep 16 19:34:04.280919 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:04.280884    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"]
Sep 16 19:34:04.289932 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:04.289901    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-whqkl"]
Sep 16 19:34:04.704481 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:04.704442    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb30978c-72b5-4f46-a040-58ad9dcce859" path="/var/lib/kubelet/pods/bb30978c-72b5-4f46-a040-58ad9dcce859/volumes"
Sep 16 19:34:09.017141 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:09.017106    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:09.039520 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:09.039481    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"]
Sep 16 19:34:09.119539 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:09.119503    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:11.849854 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:11.849816    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:11.942082 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:11.942048    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:12.124525 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:12.124431    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:12.222741 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:12.222703    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:16.050600 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:16.050532    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:16.781467 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:16.781421    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2" podUID="f7f3b533-f0b2-4c99-93d1-eb47070a29e0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:16.809814 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:16.809768    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"]
Sep 16 19:34:16.904957 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:16.904910    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:19.076370 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:19.076318    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:24.063033 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:24.062985    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2" podUID="f7f3b533-f0b2-4c99-93d1-eb47070a29e0" containerName="main" containerID="cri-o://570efe446205c4dc08553b7badc7c6f0786081293b9d71e3f87609cba60cc929" gracePeriod=45
Sep 16 19:34:25.541528 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:25.541490    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:26.781593 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:26.781538    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2" podUID="f7f3b533-f0b2-4c99-93d1-eb47070a29e0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:31.830234 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:31.830178    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm" podUID="c02f7f74-85ef-47c7-b5cf-e5d321ebdd86" containerName="main" containerID="cri-o://8eeb54ec20b9738eaf209a567fa30408b5c4c323828b1a64c277de5870f0ed86" gracePeriod=45
Sep 16 19:34:31.978455 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:31.978415    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm" podUID="c02f7f74-85ef-47c7-b5cf-e5d321ebdd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 16 19:34:32.083242 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.083184    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:34:32.201200 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.201164    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-model-cache\") pod \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") "
Sep 16 19:34:32.201383 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.201225    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-dshm\") pod \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") "
Sep 16 19:34:32.201383 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.201248    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-tmp-dir\") pod \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") "
Sep 16 19:34:32.201383 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.201292    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-tls-certs\") pod \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") "
Sep 16 19:34:32.201383 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.201377    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-home\") pod \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") "
Sep 16 19:34:32.201608 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.201420    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqmrt\" (UniqueName: \"kubernetes.io/projected/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-kube-api-access-kqmrt\") pod \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\" (UID: \"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86\") "
Sep 16 19:34:32.201608 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.201463    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-model-cache" (OuterVolumeSpecName: "model-cache") pod "c02f7f74-85ef-47c7-b5cf-e5d321ebdd86" (UID: "c02f7f74-85ef-47c7-b5cf-e5d321ebdd86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:32.201608 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.201496    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c02f7f74-85ef-47c7-b5cf-e5d321ebdd86" (UID: "c02f7f74-85ef-47c7-b5cf-e5d321ebdd86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:32.201807 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.201638    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-home" (OuterVolumeSpecName: "home") pod "c02f7f74-85ef-47c7-b5cf-e5d321ebdd86" (UID: "c02f7f74-85ef-47c7-b5cf-e5d321ebdd86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:32.201807 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.201738    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:34:32.201807 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.201755    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:34:32.201807 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.201765    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:34:32.203529 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.203504    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-dshm" (OuterVolumeSpecName: "dshm") pod "c02f7f74-85ef-47c7-b5cf-e5d321ebdd86" (UID: "c02f7f74-85ef-47c7-b5cf-e5d321ebdd86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:32.203730 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.203712    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c02f7f74-85ef-47c7-b5cf-e5d321ebdd86" (UID: "c02f7f74-85ef-47c7-b5cf-e5d321ebdd86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:32.203836 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.203816    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-kube-api-access-kqmrt" (OuterVolumeSpecName: "kube-api-access-kqmrt") pod "c02f7f74-85ef-47c7-b5cf-e5d321ebdd86" (UID: "c02f7f74-85ef-47c7-b5cf-e5d321ebdd86"). InnerVolumeSpecName "kube-api-access-kqmrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:32.302291 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.302256    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqmrt\" (UniqueName: \"kubernetes.io/projected/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-kube-api-access-kqmrt\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:34:32.302291 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.302285    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:34:32.302291 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.302294    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:34:32.400006 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.399929    2582 generic.go:356] "Generic (PLEG): container finished" podID="c02f7f74-85ef-47c7-b5cf-e5d321ebdd86" containerID="8eeb54ec20b9738eaf209a567fa30408b5c4c323828b1a64c277de5870f0ed86" exitCode=0
Sep 16 19:34:32.400006 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.399978    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"
Sep 16 19:34:32.400190 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.400040    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm" event={"ID":"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86","Type":"ContainerDied","Data":"8eeb54ec20b9738eaf209a567fa30408b5c4c323828b1a64c277de5870f0ed86"}
Sep 16 19:34:32.400190 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.400080    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm" event={"ID":"c02f7f74-85ef-47c7-b5cf-e5d321ebdd86","Type":"ContainerDied","Data":"06b00855549805f5fbc313ef17a0a572d690c586ce661bf83cc7925f25d6fe9b"}
Sep 16 19:34:32.400190 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.400100    2582 scope.go:117] "RemoveContainer" containerID="8eeb54ec20b9738eaf209a567fa30408b5c4c323828b1a64c277de5870f0ed86"
Sep 16 19:34:32.410017 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.409997    2582 scope.go:117] "RemoveContainer" containerID="8eeb54ec20b9738eaf209a567fa30408b5c4c323828b1a64c277de5870f0ed86"
Sep 16 19:34:32.410296 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:34:32.410275    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eeb54ec20b9738eaf209a567fa30408b5c4c323828b1a64c277de5870f0ed86\": container with ID starting with 8eeb54ec20b9738eaf209a567fa30408b5c4c323828b1a64c277de5870f0ed86 not found: ID does not exist" containerID="8eeb54ec20b9738eaf209a567fa30408b5c4c323828b1a64c277de5870f0ed86"
Sep 16 19:34:32.410343 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.410304    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eeb54ec20b9738eaf209a567fa30408b5c4c323828b1a64c277de5870f0ed86"} err="failed to get container status \"8eeb54ec20b9738eaf209a567fa30408b5c4c323828b1a64c277de5870f0ed86\": rpc error: code = NotFound desc = could not find container \"8eeb54ec20b9738eaf209a567fa30408b5c4c323828b1a64c277de5870f0ed86\": container with ID starting with 8eeb54ec20b9738eaf209a567fa30408b5c4c323828b1a64c277de5870f0ed86 not found: ID does not exist"
Sep 16 19:34:32.425725 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.425676    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"]
Sep 16 19:34:32.429419 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.429385    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-5zwgm"]
Sep 16 19:34:32.705101 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:32.705020    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c02f7f74-85ef-47c7-b5cf-e5d321ebdd86" path="/var/lib/kubelet/pods/c02f7f74-85ef-47c7-b5cf-e5d321ebdd86/volumes"
Sep 16 19:34:36.781095 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:36.781052    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2" podUID="f7f3b533-f0b2-4c99-93d1-eb47070a29e0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:36.781511 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:36.781161    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"
Sep 16 19:34:46.781962 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:46.781909    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2" podUID="f7f3b533-f0b2-4c99-93d1-eb47070a29e0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:54.115698 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.115654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:54.325330 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.325246    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"
Sep 16 19:34:54.416948 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.416891    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f3b533-f0b2-4c99-93d1-eb47070a29e0-tls-certs\") pod \"f7f3b533-f0b2-4c99-93d1-eb47070a29e0\" (UID: \"f7f3b533-f0b2-4c99-93d1-eb47070a29e0\") "
Sep 16 19:34:54.416948 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.416955    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h97xr\" (UniqueName: \"kubernetes.io/projected/f7f3b533-f0b2-4c99-93d1-eb47070a29e0-kube-api-access-h97xr\") pod \"f7f3b533-f0b2-4c99-93d1-eb47070a29e0\" (UID: \"f7f3b533-f0b2-4c99-93d1-eb47070a29e0\") "
Sep 16 19:34:54.419759 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.419720    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f3b533-f0b2-4c99-93d1-eb47070a29e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7f3b533-f0b2-4c99-93d1-eb47070a29e0" (UID: "f7f3b533-f0b2-4c99-93d1-eb47070a29e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:54.419893 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.419790    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f3b533-f0b2-4c99-93d1-eb47070a29e0-kube-api-access-h97xr" (OuterVolumeSpecName: "kube-api-access-h97xr") pod "f7f3b533-f0b2-4c99-93d1-eb47070a29e0" (UID: "f7f3b533-f0b2-4c99-93d1-eb47070a29e0"). InnerVolumeSpecName "kube-api-access-h97xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:54.514423 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.514370    2582 generic.go:356] "Generic (PLEG): container finished" podID="f7f3b533-f0b2-4c99-93d1-eb47070a29e0" containerID="570efe446205c4dc08553b7badc7c6f0786081293b9d71e3f87609cba60cc929" exitCode=0
Sep 16 19:34:54.514696 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.514432    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"
Sep 16 19:34:54.514696 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.514457    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2" event={"ID":"f7f3b533-f0b2-4c99-93d1-eb47070a29e0","Type":"ContainerDied","Data":"570efe446205c4dc08553b7badc7c6f0786081293b9d71e3f87609cba60cc929"}
Sep 16 19:34:54.514696 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.514519    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2" event={"ID":"f7f3b533-f0b2-4c99-93d1-eb47070a29e0","Type":"ContainerDied","Data":"197edf3340018147fbb77c1815803cd64ef0029d7d98b1e97db50d3bda340393"}
Sep 16 19:34:54.514696 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.514545    2582 scope.go:117] "RemoveContainer" containerID="570efe446205c4dc08553b7badc7c6f0786081293b9d71e3f87609cba60cc929"
Sep 16 19:34:54.517606 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.517556    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f3b533-f0b2-4c99-93d1-eb47070a29e0-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:34:54.517747 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.517609    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h97xr\" (UniqueName: \"kubernetes.io/projected/f7f3b533-f0b2-4c99-93d1-eb47070a29e0-kube-api-access-h97xr\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:34:54.525112 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.525090    2582 scope.go:117] "RemoveContainer" containerID="570efe446205c4dc08553b7badc7c6f0786081293b9d71e3f87609cba60cc929"
Sep 16 19:34:54.525438 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:34:54.525418    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570efe446205c4dc08553b7badc7c6f0786081293b9d71e3f87609cba60cc929\": container with ID starting with 570efe446205c4dc08553b7badc7c6f0786081293b9d71e3f87609cba60cc929 not found: ID does not exist" containerID="570efe446205c4dc08553b7badc7c6f0786081293b9d71e3f87609cba60cc929"
Sep 16 19:34:54.525478 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.525452    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570efe446205c4dc08553b7badc7c6f0786081293b9d71e3f87609cba60cc929"} err="failed to get container status \"570efe446205c4dc08553b7badc7c6f0786081293b9d71e3f87609cba60cc929\": rpc error: code = NotFound desc = could not find container \"570efe446205c4dc08553b7badc7c6f0786081293b9d71e3f87609cba60cc929\": container with ID starting with 570efe446205c4dc08553b7badc7c6f0786081293b9d71e3f87609cba60cc929 not found: ID does not exist"
Sep 16 19:34:54.539650 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.539620    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"]
Sep 16 19:34:54.543013 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.542984    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schekbfp2"]
Sep 16 19:34:54.703458 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:54.703371    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7f3b533-f0b2-4c99-93d1-eb47070a29e0" path="/var/lib/kubelet/pods/f7f3b533-f0b2-4c99-93d1-eb47070a29e0/volumes"
Sep 16 19:34:59.032673 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:59.032631    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:59.125978 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:34:59.125945    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:00.331883 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:00.331844    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:00.436521 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:00.436481    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:05.395162 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:05.395071    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:07.014659 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.014613    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"]
Sep 16 19:35:07.015186 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.015002    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb30978c-72b5-4f46-a040-58ad9dcce859" containerName="main"
Sep 16 19:35:07.015186 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.015016    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb30978c-72b5-4f46-a040-58ad9dcce859" containerName="main"
Sep 16 19:35:07.015186 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.015037    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f3b533-f0b2-4c99-93d1-eb47070a29e0" containerName="main"
Sep 16 19:35:07.015186 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.015043    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f3b533-f0b2-4c99-93d1-eb47070a29e0" containerName="main"
Sep 16 19:35:07.015186 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.015057    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c02f7f74-85ef-47c7-b5cf-e5d321ebdd86" containerName="main"
Sep 16 19:35:07.015186 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.015063    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c02f7f74-85ef-47c7-b5cf-e5d321ebdd86" containerName="main"
Sep 16 19:35:07.015186 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.015123    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f3b533-f0b2-4c99-93d1-eb47070a29e0" containerName="main"
Sep 16 19:35:07.015186 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.015130    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c02f7f74-85ef-47c7-b5cf-e5d321ebdd86" containerName="main"
Sep 16 19:35:07.015186 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.015138    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb30978c-72b5-4f46-a040-58ad9dcce859" containerName="main"
Sep 16 19:35:07.020198 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.020170    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.023485 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.023458    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 19:35:07.023911 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.023873    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 16 19:35:07.023911 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.023903    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:35:07.024101 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.023920    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-kzmld\""
Sep 16 19:35:07.024101 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.023997    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 19:35:07.024101 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.024047    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 19:35:07.037646 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.037615    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"]
Sep 16 19:35:07.040238 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.040210    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:07.144847 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.144795    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm8ps\" (UniqueName: \"kubernetes.io/projected/7ff7b651-04d1-410c-a027-cccef38a6e48-kube-api-access-vm8ps\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.144847 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.144848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.145161 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.144880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ff7b651-04d1-410c-a027-cccef38a6e48-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.145161 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.144932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.145161 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.144990    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.145161 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.145057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.145161 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.145090    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff7b651-04d1-410c-a027-cccef38a6e48-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.145161 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.145138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.228073 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.228031    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"]
Sep 16 19:35:07.234276 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.234241    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"
Sep 16 19:35:07.238017 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.237972    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-2rp9m\""
Sep 16 19:35:07.239430 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.239400    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"]
Sep 16 19:35:07.246675 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.246645    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm8ps\" (UniqueName: \"kubernetes.io/projected/7ff7b651-04d1-410c-a027-cccef38a6e48-kube-api-access-vm8ps\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.246847 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.246696    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.246847 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.246740    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ff7b651-04d1-410c-a027-cccef38a6e48-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.246847 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.246806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.246847 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.246835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.247088 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.246881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.247088 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.246914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff7b651-04d1-410c-a027-cccef38a6e48-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.247088 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.246946    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.247505 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.247337    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.247505 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.247390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.247706 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.247604    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.247706 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.247660    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.248350 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.248327    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ff7b651-04d1-410c-a027-cccef38a6e48-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.250223 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.250191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.250667 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.250644    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff7b651-04d1-410c-a027-cccef38a6e48-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.260301 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.260276    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm8ps\" (UniqueName: \"kubernetes.io/projected/7ff7b651-04d1-410c-a027-cccef38a6e48-kube-api-access-vm8ps\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.264794 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.264718    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:07.332510 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.332453    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:07.348158 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.348125    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2368ffbf-8523-46a6-aaba-2143f9f7c4d9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq\" (UID: \"2368ffbf-8523-46a6-aaba-2143f9f7c4d9\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"
Sep 16 19:35:07.348297 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.348173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdcz4\" (UniqueName: \"kubernetes.io/projected/2368ffbf-8523-46a6-aaba-2143f9f7c4d9-kube-api-access-xdcz4\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq\" (UID: \"2368ffbf-8523-46a6-aaba-2143f9f7c4d9\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"
Sep 16 19:35:07.449225 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.449187    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2368ffbf-8523-46a6-aaba-2143f9f7c4d9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq\" (UID: \"2368ffbf-8523-46a6-aaba-2143f9f7c4d9\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"
Sep 16 19:35:07.449429 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.449230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdcz4\" (UniqueName: \"kubernetes.io/projected/2368ffbf-8523-46a6-aaba-2143f9f7c4d9-kube-api-access-xdcz4\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq\" (UID: \"2368ffbf-8523-46a6-aaba-2143f9f7c4d9\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"
Sep 16 19:35:07.452154 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.452121    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2368ffbf-8523-46a6-aaba-2143f9f7c4d9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq\" (UID: \"2368ffbf-8523-46a6-aaba-2143f9f7c4d9\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"
Sep 16 19:35:07.458334 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.458306    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdcz4\" (UniqueName: \"kubernetes.io/projected/2368ffbf-8523-46a6-aaba-2143f9f7c4d9-kube-api-access-xdcz4\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq\" (UID: \"2368ffbf-8523-46a6-aaba-2143f9f7c4d9\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"
Sep 16 19:35:07.551034 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.550971    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"
Sep 16 19:35:07.615137 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.615103    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"]
Sep 16 19:35:07.619781 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:35:07.619730    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff7b651_04d1_410c_a027_cccef38a6e48.slice/crio-e10cc363ba392ffe522e1d1fd1029dc3cb3d9257c631c972323937434d2e5b89 WatchSource:0}: Error finding container e10cc363ba392ffe522e1d1fd1029dc3cb3d9257c631c972323937434d2e5b89: Status 404 returned error can't find the container with id e10cc363ba392ffe522e1d1fd1029dc3cb3d9257c631c972323937434d2e5b89
Sep 16 19:35:07.836376 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:07.836342    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"]
Sep 16 19:35:07.839977 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:35:07.839925    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2368ffbf_8523_46a6_aaba_2143f9f7c4d9.slice/crio-1f7b7888518913f31bfb2e9de7e59d7be5a35ebd6c03bf93622062c1c4692e6a WatchSource:0}: Error finding container 1f7b7888518913f31bfb2e9de7e59d7be5a35ebd6c03bf93622062c1c4692e6a: Status 404 returned error can't find the container with id 1f7b7888518913f31bfb2e9de7e59d7be5a35ebd6c03bf93622062c1c4692e6a
Sep 16 19:35:08.586068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:08.586005    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq" event={"ID":"2368ffbf-8523-46a6-aaba-2143f9f7c4d9","Type":"ContainerStarted","Data":"abf92f4d85072a2aafe330073cc961bd248d9fa867b4c4a4fcca261ba6958a17"}
Sep 16 19:35:08.586723 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:08.586094    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq" event={"ID":"2368ffbf-8523-46a6-aaba-2143f9f7c4d9","Type":"ContainerStarted","Data":"1f7b7888518913f31bfb2e9de7e59d7be5a35ebd6c03bf93622062c1c4692e6a"}
Sep 16 19:35:08.586723 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:08.586129    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"
Sep 16 19:35:08.587959 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:08.587928    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" event={"ID":"7ff7b651-04d1-410c-a027-cccef38a6e48","Type":"ContainerStarted","Data":"1949ca47562bcb9852d7d6bd19cb874002213d3e6db34c9d9802ce8131999991"}
Sep 16 19:35:08.588088 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:08.587979    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" event={"ID":"7ff7b651-04d1-410c-a027-cccef38a6e48","Type":"ContainerStarted","Data":"e10cc363ba392ffe522e1d1fd1029dc3cb3d9257c631c972323937434d2e5b89"}
Sep 16 19:35:08.604145 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:08.604054    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq" podStartSLOduration=1.604026017 podStartE2EDuration="1.604026017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:08.603827755 +0000 UTC m=+2966.434436324" watchObservedRunningTime="2026-09-16 19:35:08.604026017 +0000 UTC m=+2966.434634563"
Sep 16 19:35:11.136445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:11.136393    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:11.203083 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:11.202971    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:11.605507 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:11.605457    2582 generic.go:356] "Generic (PLEG): container finished" podID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerID="1949ca47562bcb9852d7d6bd19cb874002213d3e6db34c9d9802ce8131999991" exitCode=0
Sep 16 19:35:11.605798 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:11.605532    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" event={"ID":"7ff7b651-04d1-410c-a027-cccef38a6e48","Type":"ContainerDied","Data":"1949ca47562bcb9852d7d6bd19cb874002213d3e6db34c9d9802ce8131999991"}
Sep 16 19:35:12.613094 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:12.613040    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" event={"ID":"7ff7b651-04d1-410c-a027-cccef38a6e48","Type":"ContainerStarted","Data":"8ad01e27534e57d540e8afcdfd025bc976bcb72f39f42523d410172a1ffe00ba"}
Sep 16 19:35:12.641273 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:12.641201    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" podStartSLOduration=3.445108856 podStartE2EDuration="6.641181106s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.19607225s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:12.637845746 +0000 UTC m=+2970.468454313" watchObservedRunningTime="2026-09-16 19:35:12.641181106 +0000 UTC m=+2970.471789650"
Sep 16 19:35:13.333663 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:13.333614    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:17.332701 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:17.332631    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:35:17.334765 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:17.334732    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 19:35:19.972725 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:19.972682    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:27.333701 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:27.333636    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 19:35:37.333848 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:37.333788    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 19:35:39.595194 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:39.595157    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"
Sep 16 19:35:44.476178 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:44.476119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:47.333114 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:47.333060    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 19:35:49.485080 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:49.485037    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:49.578477 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:49.578437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:54.473306 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:54.473266    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:57.333364 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:35:57.333298    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 19:36:03.636988 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:03.636942    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:04.249844 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:04.249794    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:07.333323 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:07.333270    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 19:36:15.935714 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:15.935628    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:16.761738 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:16.761706    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:17.333353 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:17.333315    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 19:36:22.440425 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:22.440391    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:27.333161 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:27.333106    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 19:36:37.226551 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:37.226522    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:37.343802 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:37.343767    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:36:37.351510 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:37.351483    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:36:45.321514 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:45.321476    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:45.383319 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:45.383282    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:50.321651 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:50.321617    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:54.984129 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:54.984084    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"]
Sep 16 19:36:54.986057 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:54.986007    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"]
Sep 16 19:36:54.989518 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:54.989426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:55.056051 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:55.056001    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:59.595422 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:36:59.595372    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq" podUID="2368ffbf-8523-46a6-aaba-2143f9f7c4d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:09.594926 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:09.594882    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq" podUID="2368ffbf-8523-46a6-aaba-2143f9f7c4d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:10.006411 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.006279    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq" podUID="2368ffbf-8523-46a6-aaba-2143f9f7c4d9" containerName="main" containerID="cri-o://abf92f4d85072a2aafe330073cc961bd248d9fa867b4c4a4fcca261ba6958a17" gracePeriod=45
Sep 16 19:37:10.009229 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.009153    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="main" containerID="cri-o://8ad01e27534e57d540e8afcdfd025bc976bcb72f39f42523d410172a1ffe00ba" gracePeriod=45
Sep 16 19:37:10.345472 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.345415    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 16 19:37:10.657403 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.657371    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:37:10.815639 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.815606    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ff7b651-04d1-410c-a027-cccef38a6e48-cabundle-cert\") pod \"7ff7b651-04d1-410c-a027-cccef38a6e48\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") "
Sep 16 19:37:10.815845 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.815721    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-tmp-dir\") pod \"7ff7b651-04d1-410c-a027-cccef38a6e48\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") "
Sep 16 19:37:10.815845 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.815749    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-home\") pod \"7ff7b651-04d1-410c-a027-cccef38a6e48\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") "
Sep 16 19:37:10.815845 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.815768    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff7b651-04d1-410c-a027-cccef38a6e48-tls-certs\") pod \"7ff7b651-04d1-410c-a027-cccef38a6e48\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") "
Sep 16 19:37:10.815845 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.815795    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-kserve-provision-location\") pod \"7ff7b651-04d1-410c-a027-cccef38a6e48\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") "
Sep 16 19:37:10.815845 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.815827    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-dshm\") pod \"7ff7b651-04d1-410c-a027-cccef38a6e48\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") "
Sep 16 19:37:10.816121 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.815869    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm8ps\" (UniqueName: \"kubernetes.io/projected/7ff7b651-04d1-410c-a027-cccef38a6e48-kube-api-access-vm8ps\") pod \"7ff7b651-04d1-410c-a027-cccef38a6e48\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") "
Sep 16 19:37:10.816121 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.815902    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-model-cache\") pod \"7ff7b651-04d1-410c-a027-cccef38a6e48\" (UID: \"7ff7b651-04d1-410c-a027-cccef38a6e48\") "
Sep 16 19:37:10.816234 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.816058    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff7b651-04d1-410c-a027-cccef38a6e48-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7ff7b651-04d1-410c-a027-cccef38a6e48" (UID: "7ff7b651-04d1-410c-a027-cccef38a6e48"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:37:10.816290 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.816247    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ff7b651-04d1-410c-a027-cccef38a6e48-cabundle-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:37:10.816290 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.816270    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-home" (OuterVolumeSpecName: "home") pod "7ff7b651-04d1-410c-a027-cccef38a6e48" (UID: "7ff7b651-04d1-410c-a027-cccef38a6e48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:10.816478 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.816390    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ff7b651-04d1-410c-a027-cccef38a6e48" (UID: "7ff7b651-04d1-410c-a027-cccef38a6e48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:10.816687 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.816667    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ff7b651-04d1-410c-a027-cccef38a6e48" (UID: "7ff7b651-04d1-410c-a027-cccef38a6e48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:10.818608 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.818561    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff7b651-04d1-410c-a027-cccef38a6e48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ff7b651-04d1-410c-a027-cccef38a6e48" (UID: "7ff7b651-04d1-410c-a027-cccef38a6e48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:37:10.818739 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.818601    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-dshm" (OuterVolumeSpecName: "dshm") pod "7ff7b651-04d1-410c-a027-cccef38a6e48" (UID: "7ff7b651-04d1-410c-a027-cccef38a6e48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:10.818806 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.818782    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff7b651-04d1-410c-a027-cccef38a6e48-kube-api-access-vm8ps" (OuterVolumeSpecName: "kube-api-access-vm8ps") pod "7ff7b651-04d1-410c-a027-cccef38a6e48" (UID: "7ff7b651-04d1-410c-a027-cccef38a6e48"). InnerVolumeSpecName "kube-api-access-vm8ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:37:10.838997 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.838963    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ff7b651-04d1-410c-a027-cccef38a6e48" (UID: "7ff7b651-04d1-410c-a027-cccef38a6e48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:10.917282 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.917191    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:37:10.917282 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.917222    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:37:10.917282 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.917231    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ff7b651-04d1-410c-a027-cccef38a6e48-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:37:10.917282 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.917242    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:37:10.917282 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.917251    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:37:10.917282 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.917259    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm8ps\" (UniqueName: \"kubernetes.io/projected/7ff7b651-04d1-410c-a027-cccef38a6e48-kube-api-access-vm8ps\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:37:10.917282 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:10.917267    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ff7b651-04d1-410c-a027-cccef38a6e48-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:37:11.220040 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:11.219938    2582 generic.go:356] "Generic (PLEG): container finished" podID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerID="8ad01e27534e57d540e8afcdfd025bc976bcb72f39f42523d410172a1ffe00ba" exitCode=0
Sep 16 19:37:11.220040 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:11.219986    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" event={"ID":"7ff7b651-04d1-410c-a027-cccef38a6e48","Type":"ContainerDied","Data":"8ad01e27534e57d540e8afcdfd025bc976bcb72f39f42523d410172a1ffe00ba"}
Sep 16 19:37:11.220040 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:11.220008    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"
Sep 16 19:37:11.220040 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:11.220032    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq" event={"ID":"7ff7b651-04d1-410c-a027-cccef38a6e48","Type":"ContainerDied","Data":"e10cc363ba392ffe522e1d1fd1029dc3cb3d9257c631c972323937434d2e5b89"}
Sep 16 19:37:11.220333 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:11.220056    2582 scope.go:117] "RemoveContainer" containerID="8ad01e27534e57d540e8afcdfd025bc976bcb72f39f42523d410172a1ffe00ba"
Sep 16 19:37:11.230737 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:11.230642    2582 scope.go:117] "RemoveContainer" containerID="1949ca47562bcb9852d7d6bd19cb874002213d3e6db34c9d9802ce8131999991"
Sep 16 19:37:11.239851 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:11.239828    2582 scope.go:117] "RemoveContainer" containerID="8ad01e27534e57d540e8afcdfd025bc976bcb72f39f42523d410172a1ffe00ba"
Sep 16 19:37:11.240167 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:37:11.240143    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad01e27534e57d540e8afcdfd025bc976bcb72f39f42523d410172a1ffe00ba\": container with ID starting with 8ad01e27534e57d540e8afcdfd025bc976bcb72f39f42523d410172a1ffe00ba not found: ID does not exist" containerID="8ad01e27534e57d540e8afcdfd025bc976bcb72f39f42523d410172a1ffe00ba"
Sep 16 19:37:11.240250 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:11.240178    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad01e27534e57d540e8afcdfd025bc976bcb72f39f42523d410172a1ffe00ba"} err="failed to get container status \"8ad01e27534e57d540e8afcdfd025bc976bcb72f39f42523d410172a1ffe00ba\": rpc error: code = NotFound desc = could not find container \"8ad01e27534e57d540e8afcdfd025bc976bcb72f39f42523d410172a1ffe00ba\": container with ID starting with 8ad01e27534e57d540e8afcdfd025bc976bcb72f39f42523d410172a1ffe00ba not found: ID does not exist"
Sep 16 19:37:11.240250 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:11.240200    2582 scope.go:117] "RemoveContainer" containerID="1949ca47562bcb9852d7d6bd19cb874002213d3e6db34c9d9802ce8131999991"
Sep 16 19:37:11.240506 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:37:11.240488    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1949ca47562bcb9852d7d6bd19cb874002213d3e6db34c9d9802ce8131999991\": container with ID starting with 1949ca47562bcb9852d7d6bd19cb874002213d3e6db34c9d9802ce8131999991 not found: ID does not exist" containerID="1949ca47562bcb9852d7d6bd19cb874002213d3e6db34c9d9802ce8131999991"
Sep 16 19:37:11.240548 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:11.240513    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1949ca47562bcb9852d7d6bd19cb874002213d3e6db34c9d9802ce8131999991"} err="failed to get container status \"1949ca47562bcb9852d7d6bd19cb874002213d3e6db34c9d9802ce8131999991\": rpc error: code = NotFound desc = could not find container \"1949ca47562bcb9852d7d6bd19cb874002213d3e6db34c9d9802ce8131999991\": container with ID starting with 1949ca47562bcb9852d7d6bd19cb874002213d3e6db34c9d9802ce8131999991 not found: ID does not exist"
Sep 16 19:37:11.244546 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:11.244513    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"]
Sep 16 19:37:11.252308 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:11.252279    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-785788c6-8vhzq"]
Sep 16 19:37:12.704709 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:12.704673    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" path="/var/lib/kubelet/pods/7ff7b651-04d1-410c-a027-cccef38a6e48/volumes"
Sep 16 19:37:19.594627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:19.594557    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq" podUID="2368ffbf-8523-46a6-aaba-2143f9f7c4d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:19.595061 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:19.594696    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"
Sep 16 19:37:29.594653 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:29.594612    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq" podUID="2368ffbf-8523-46a6-aaba-2143f9f7c4d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:39.595059 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:39.595019    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq" podUID="2368ffbf-8523-46a6-aaba-2143f9f7c4d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:40.256837 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.256807    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"
Sep 16 19:37:40.350875 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.350836    2582 generic.go:356] "Generic (PLEG): container finished" podID="2368ffbf-8523-46a6-aaba-2143f9f7c4d9" containerID="abf92f4d85072a2aafe330073cc961bd248d9fa867b4c4a4fcca261ba6958a17" exitCode=0
Sep 16 19:37:40.351070 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.350897    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"
Sep 16 19:37:40.351070 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.350922    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq" event={"ID":"2368ffbf-8523-46a6-aaba-2143f9f7c4d9","Type":"ContainerDied","Data":"abf92f4d85072a2aafe330073cc961bd248d9fa867b4c4a4fcca261ba6958a17"}
Sep 16 19:37:40.351070 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.350985    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq" event={"ID":"2368ffbf-8523-46a6-aaba-2143f9f7c4d9","Type":"ContainerDied","Data":"1f7b7888518913f31bfb2e9de7e59d7be5a35ebd6c03bf93622062c1c4692e6a"}
Sep 16 19:37:40.351070 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.351019    2582 scope.go:117] "RemoveContainer" containerID="abf92f4d85072a2aafe330073cc961bd248d9fa867b4c4a4fcca261ba6958a17"
Sep 16 19:37:40.360022 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.360002    2582 scope.go:117] "RemoveContainer" containerID="abf92f4d85072a2aafe330073cc961bd248d9fa867b4c4a4fcca261ba6958a17"
Sep 16 19:37:40.360338 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:37:40.360318    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf92f4d85072a2aafe330073cc961bd248d9fa867b4c4a4fcca261ba6958a17\": container with ID starting with abf92f4d85072a2aafe330073cc961bd248d9fa867b4c4a4fcca261ba6958a17 not found: ID does not exist" containerID="abf92f4d85072a2aafe330073cc961bd248d9fa867b4c4a4fcca261ba6958a17"
Sep 16 19:37:40.360389 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.360348    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf92f4d85072a2aafe330073cc961bd248d9fa867b4c4a4fcca261ba6958a17"} err="failed to get container status \"abf92f4d85072a2aafe330073cc961bd248d9fa867b4c4a4fcca261ba6958a17\": rpc error: code = NotFound desc = could not find container \"abf92f4d85072a2aafe330073cc961bd248d9fa867b4c4a4fcca261ba6958a17\": container with ID starting with abf92f4d85072a2aafe330073cc961bd248d9fa867b4c4a4fcca261ba6958a17 not found: ID does not exist"
Sep 16 19:37:40.380540 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.380511    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdcz4\" (UniqueName: \"kubernetes.io/projected/2368ffbf-8523-46a6-aaba-2143f9f7c4d9-kube-api-access-xdcz4\") pod \"2368ffbf-8523-46a6-aaba-2143f9f7c4d9\" (UID: \"2368ffbf-8523-46a6-aaba-2143f9f7c4d9\") "
Sep 16 19:37:40.380679 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.380626    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2368ffbf-8523-46a6-aaba-2143f9f7c4d9-tls-certs\") pod \"2368ffbf-8523-46a6-aaba-2143f9f7c4d9\" (UID: \"2368ffbf-8523-46a6-aaba-2143f9f7c4d9\") "
Sep 16 19:37:40.382932 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.382894    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2368ffbf-8523-46a6-aaba-2143f9f7c4d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2368ffbf-8523-46a6-aaba-2143f9f7c4d9" (UID: "2368ffbf-8523-46a6-aaba-2143f9f7c4d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:37:40.382932 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.382925    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2368ffbf-8523-46a6-aaba-2143f9f7c4d9-kube-api-access-xdcz4" (OuterVolumeSpecName: "kube-api-access-xdcz4") pod "2368ffbf-8523-46a6-aaba-2143f9f7c4d9" (UID: "2368ffbf-8523-46a6-aaba-2143f9f7c4d9"). InnerVolumeSpecName "kube-api-access-xdcz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:37:40.482031 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.481992    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2368ffbf-8523-46a6-aaba-2143f9f7c4d9-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:37:40.482031 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.482028    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdcz4\" (UniqueName: \"kubernetes.io/projected/2368ffbf-8523-46a6-aaba-2143f9f7c4d9-kube-api-access-xdcz4\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:37:40.673909 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.673876    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"]
Sep 16 19:37:40.677856 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.677820    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheg46dq"]
Sep 16 19:37:40.704068 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:40.704031    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2368ffbf-8523-46a6-aaba-2143f9f7c4d9" path="/var/lib/kubelet/pods/2368ffbf-8523-46a6-aaba-2143f9f7c4d9/volumes"
Sep 16 19:37:43.413497 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:43.413465    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:43.517252 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:43.517216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:43.613820 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:43.613784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:45.675504 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:45.675469    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:45.775521 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:45.775481    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:45.871745 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:45.871715    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:49.399833 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:49.399800    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:51.391418 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:51.391381    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:51.454179 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:51.454138    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:54.913641 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:54.913600    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:55.006276 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:55.006227    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:58.423296 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.423241    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"]
Sep 16 19:37:58.423822 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.423804    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="main"
Sep 16 19:37:58.423866 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.423827    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="main"
Sep 16 19:37:58.423866 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.423863    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2368ffbf-8523-46a6-aaba-2143f9f7c4d9" containerName="main"
Sep 16 19:37:58.424189 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.423871    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2368ffbf-8523-46a6-aaba-2143f9f7c4d9" containerName="main"
Sep 16 19:37:58.424189 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.423884    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="storage-initializer"
Sep 16 19:37:58.424189 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.423894    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="storage-initializer"
Sep 16 19:37:58.424189 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.423993    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ff7b651-04d1-410c-a027-cccef38a6e48" containerName="main"
Sep 16 19:37:58.424189 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.424008    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2368ffbf-8523-46a6-aaba-2143f9f7c4d9" containerName="main"
Sep 16 19:37:58.426699 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.426678    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.430230 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.430201    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 16 19:37:58.430741 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.430711    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 16 19:37:58.430741 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.430753    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-7f7pj\""
Sep 16 19:37:58.431000 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.430823    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 16 19:37:58.432329 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.432291    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 16 19:37:58.444840 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.444807    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"]
Sep 16 19:37:58.459850 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.459805    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:58.540202 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.540164    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-tls-certs\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.540202 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.540207    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.540445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.540227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c22d7\" (UniqueName: \"kubernetes.io/projected/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-kube-api-access-c22d7\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.540445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.540249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.540445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.540349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-model-cache\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.540445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.540387    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-dshm\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.540593 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.540456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-home\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.641529 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.641489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-model-cache\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.641529 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.641532    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-dshm\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.641764 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.641603    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-home\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.641764 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.641658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-tls-certs\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.641764 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.641698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.641764 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.641738    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c22d7\" (UniqueName: \"kubernetes.io/projected/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-kube-api-access-c22d7\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.641968 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.641827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.641968 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.641951    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-model-cache\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.642162 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.642137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.642243 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.642198    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.642287 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.642255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-home\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.644715 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.644698    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-dshm\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.645114 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.645091    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-tls-certs\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.651994 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.651959    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c22d7\" (UniqueName: \"kubernetes.io/projected/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-kube-api-access-c22d7\") pod \"e2e-del-blocked-svc-kserve-69498f7875-vrr6l\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.687828 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.687749    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"]
Sep 16 19:37:58.690671 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.690647    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"
Sep 16 19:37:58.694041 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.694010    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-g67f6\""
Sep 16 19:37:58.706408 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.706376    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"]
Sep 16 19:37:58.739370 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.739330    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:37:58.740562 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.740531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:58.844303 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.844252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwqsn\" (UniqueName: \"kubernetes.io/projected/391953c9-797b-47a3-9d49-b842fa3197f8-kube-api-access-wwqsn\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds\" (UID: \"391953c9-797b-47a3-9d49-b842fa3197f8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"
Sep 16 19:37:58.844545 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.844475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/391953c9-797b-47a3-9d49-b842fa3197f8-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds\" (UID: \"391953c9-797b-47a3-9d49-b842fa3197f8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"
Sep 16 19:37:58.945333 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.945207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/391953c9-797b-47a3-9d49-b842fa3197f8-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds\" (UID: \"391953c9-797b-47a3-9d49-b842fa3197f8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"
Sep 16 19:37:58.945333 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.945286    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwqsn\" (UniqueName: \"kubernetes.io/projected/391953c9-797b-47a3-9d49-b842fa3197f8-kube-api-access-wwqsn\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds\" (UID: \"391953c9-797b-47a3-9d49-b842fa3197f8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"
Sep 16 19:37:58.948200 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.948168    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/391953c9-797b-47a3-9d49-b842fa3197f8-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds\" (UID: \"391953c9-797b-47a3-9d49-b842fa3197f8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"
Sep 16 19:37:58.955091 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:58.955059    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwqsn\" (UniqueName: \"kubernetes.io/projected/391953c9-797b-47a3-9d49-b842fa3197f8-kube-api-access-wwqsn\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds\" (UID: \"391953c9-797b-47a3-9d49-b842fa3197f8\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"
Sep 16 19:37:59.002958 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:59.002914    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"
Sep 16 19:37:59.021758 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:59.021727    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"]
Sep 16 19:37:59.024239 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:37:59.024205    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf3847d_1c81_45db_b9e1_f7fc38c235b6.slice/crio-49a8fb00779de413a2b03d5d25226abe37a92ec40ba0b59ea0d1f7d4b8643671 WatchSource:0}: Error finding container 49a8fb00779de413a2b03d5d25226abe37a92ec40ba0b59ea0d1f7d4b8643671: Status 404 returned error can't find the container with id 49a8fb00779de413a2b03d5d25226abe37a92ec40ba0b59ea0d1f7d4b8643671
Sep 16 19:37:59.284997 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:59.284957    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"]
Sep 16 19:37:59.286848 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:37:59.286819    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391953c9_797b_47a3_9d49_b842fa3197f8.slice/crio-e1f17ed0438af39e475a547cc95e92d428dcecb335a944b9834459238d70c8ac WatchSource:0}: Error finding container e1f17ed0438af39e475a547cc95e92d428dcecb335a944b9834459238d70c8ac: Status 404 returned error can't find the container with id e1f17ed0438af39e475a547cc95e92d428dcecb335a944b9834459238d70c8ac
Sep 16 19:37:59.452823 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:59.452779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds" event={"ID":"391953c9-797b-47a3-9d49-b842fa3197f8","Type":"ContainerStarted","Data":"5e84793e8a33d3772b6b6a7a248b3cdf25e5c547b3391a086b87640810be3cd5"}
Sep 16 19:37:59.453317 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:59.452846    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds" event={"ID":"391953c9-797b-47a3-9d49-b842fa3197f8","Type":"ContainerStarted","Data":"e1f17ed0438af39e475a547cc95e92d428dcecb335a944b9834459238d70c8ac"}
Sep 16 19:37:59.453317 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:59.452897    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"
Sep 16 19:37:59.454666 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:59.454628    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l" event={"ID":"1bf3847d-1c81-45db-b9e1-f7fc38c235b6","Type":"ContainerStarted","Data":"c87151f423b7c0b0f5171439f63751e084163a74efed297b06e157381f3ba993"}
Sep 16 19:37:59.454787 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:59.454680    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l" event={"ID":"1bf3847d-1c81-45db-b9e1-f7fc38c235b6","Type":"ContainerStarted","Data":"49a8fb00779de413a2b03d5d25226abe37a92ec40ba0b59ea0d1f7d4b8643671"}
Sep 16 19:37:59.482570 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:37:59.482478    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds" podStartSLOduration=1.482460285 podStartE2EDuration="1.482460285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:59.480565448 +0000 UTC m=+3137.311173994" watchObservedRunningTime="2026-09-16 19:37:59.482460285 +0000 UTC m=+3137.313068840"
Sep 16 19:38:01.315742 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:01.315697    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:01.505797 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:01.505761    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"]
Sep 16 19:38:01.506176 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:01.506121    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l" podUID="1bf3847d-1c81-45db-b9e1-f7fc38c235b6" containerName="storage-initializer" containerID="cri-o://c87151f423b7c0b0f5171439f63751e084163a74efed297b06e157381f3ba993" gracePeriod=60
Sep 16 19:38:01.512483 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:01.512456    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"]
Sep 16 19:38:01.514601 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:01.514486    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:02.231046 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:02.231014    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:02.333724 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:02.333691    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:02.431644 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:02.431616    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:03.859365 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.859337    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:38:03.995217 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.995127    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-home\") pod \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") "
Sep 16 19:38:03.995217 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.995198    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c22d7\" (UniqueName: \"kubernetes.io/projected/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-kube-api-access-c22d7\") pod \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") "
Sep 16 19:38:03.995449 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.995240    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-kserve-provision-location\") pod \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") "
Sep 16 19:38:03.995449 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.995295    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-model-cache\") pod \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") "
Sep 16 19:38:03.995449 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.995332    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-dshm\") pod \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") "
Sep 16 19:38:03.995449 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.995369    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-tmp-dir\") pod \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") "
Sep 16 19:38:03.995449 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.995394    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-tls-certs\") pod \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\" (UID: \"1bf3847d-1c81-45db-b9e1-f7fc38c235b6\") "
Sep 16 19:38:03.995761 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.995457    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-home" (OuterVolumeSpecName: "home") pod "1bf3847d-1c81-45db-b9e1-f7fc38c235b6" (UID: "1bf3847d-1c81-45db-b9e1-f7fc38c235b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:03.995761 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.995680    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bf3847d-1c81-45db-b9e1-f7fc38c235b6" (UID: "1bf3847d-1c81-45db-b9e1-f7fc38c235b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:03.995761 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.995700    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bf3847d-1c81-45db-b9e1-f7fc38c235b6" (UID: "1bf3847d-1c81-45db-b9e1-f7fc38c235b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:03.995761 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.995719    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:38:03.997629 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.997601    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-dshm" (OuterVolumeSpecName: "dshm") pod "1bf3847d-1c81-45db-b9e1-f7fc38c235b6" (UID: "1bf3847d-1c81-45db-b9e1-f7fc38c235b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:03.997751 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.997664    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bf3847d-1c81-45db-b9e1-f7fc38c235b6" (UID: "1bf3847d-1c81-45db-b9e1-f7fc38c235b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:03.997751 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:03.997706    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-kube-api-access-c22d7" (OuterVolumeSpecName: "kube-api-access-c22d7") pod "1bf3847d-1c81-45db-b9e1-f7fc38c235b6" (UID: "1bf3847d-1c81-45db-b9e1-f7fc38c235b6"). InnerVolumeSpecName "kube-api-access-c22d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:04.077408 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.077343    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1bf3847d-1c81-45db-b9e1-f7fc38c235b6" (UID: "1bf3847d-1c81-45db-b9e1-f7fc38c235b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:04.096749 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.096713    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:38:04.096749 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.096746    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:38:04.096749 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.096755    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:38:04.096949 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.096783    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:38:04.096949 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.096791    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:38:04.096949 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.096799    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c22d7\" (UniqueName: \"kubernetes.io/projected/1bf3847d-1c81-45db-b9e1-f7fc38c235b6-kube-api-access-c22d7\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:38:04.488511 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.488468    2582 generic.go:356] "Generic (PLEG): container finished" podID="1bf3847d-1c81-45db-b9e1-f7fc38c235b6" containerID="c87151f423b7c0b0f5171439f63751e084163a74efed297b06e157381f3ba993" exitCode=0
Sep 16 19:38:04.488693 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.488515    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"
Sep 16 19:38:04.488693 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.488554    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l" event={"ID":"1bf3847d-1c81-45db-b9e1-f7fc38c235b6","Type":"ContainerDied","Data":"c87151f423b7c0b0f5171439f63751e084163a74efed297b06e157381f3ba993"}
Sep 16 19:38:04.488693 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.488620    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l" event={"ID":"1bf3847d-1c81-45db-b9e1-f7fc38c235b6","Type":"ContainerDied","Data":"49a8fb00779de413a2b03d5d25226abe37a92ec40ba0b59ea0d1f7d4b8643671"}
Sep 16 19:38:04.488693 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.488644    2582 scope.go:117] "RemoveContainer" containerID="c87151f423b7c0b0f5171439f63751e084163a74efed297b06e157381f3ba993"
Sep 16 19:38:04.498498 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.498478    2582 scope.go:117] "RemoveContainer" containerID="c87151f423b7c0b0f5171439f63751e084163a74efed297b06e157381f3ba993"
Sep 16 19:38:04.498773 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:04.498747    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87151f423b7c0b0f5171439f63751e084163a74efed297b06e157381f3ba993\": container with ID starting with c87151f423b7c0b0f5171439f63751e084163a74efed297b06e157381f3ba993 not found: ID does not exist" containerID="c87151f423b7c0b0f5171439f63751e084163a74efed297b06e157381f3ba993"
Sep 16 19:38:04.498834 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.498775    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87151f423b7c0b0f5171439f63751e084163a74efed297b06e157381f3ba993"} err="failed to get container status \"c87151f423b7c0b0f5171439f63751e084163a74efed297b06e157381f3ba993\": rpc error: code = NotFound desc = could not find container \"c87151f423b7c0b0f5171439f63751e084163a74efed297b06e157381f3ba993\": container with ID starting with c87151f423b7c0b0f5171439f63751e084163a74efed297b06e157381f3ba993 not found: ID does not exist"
Sep 16 19:38:04.532793 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.532767    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"]
Sep 16 19:38:04.537109 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.537084    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-69498f7875-vrr6l"]
Sep 16 19:38:04.703338 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:04.703297    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bf3847d-1c81-45db-b9e1-f7fc38c235b6" path="/var/lib/kubelet/pods/1bf3847d-1c81-45db-b9e1-f7fc38c235b6/volumes"
Sep 16 19:38:06.565853 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:06.565823    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:06.668074 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:06.668034    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:06.774612 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:06.774553    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:07.302503 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:07.302475    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:09.419083 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:09.419049    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:09.511691 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:09.511664    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:11.017443 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:11.017408    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:11.280373 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:11.280302    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:11.379298 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:11.379260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:12.939292 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:12.939256    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:13.007635 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:13.007611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:16.534716 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:16.534665    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds" podUID="391953c9-797b-47a3-9d49-b842fa3197f8" containerName="main" containerID="cri-o://5e84793e8a33d3772b6b6a7a248b3cdf25e5c547b3391a086b87640810be3cd5" gracePeriod=45
Sep 16 19:38:16.710691 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:16.710653    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:16.775143 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:16.775109    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:22.597853 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.597807    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"]
Sep 16 19:38:22.599550 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.598356    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf3847d-1c81-45db-b9e1-f7fc38c235b6" containerName="storage-initializer"
Sep 16 19:38:22.599550 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.598375    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf3847d-1c81-45db-b9e1-f7fc38c235b6" containerName="storage-initializer"
Sep 16 19:38:22.599550 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.598443    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf3847d-1c81-45db-b9e1-f7fc38c235b6" containerName="storage-initializer"
Sep 16 19:38:22.600710 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.600688    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"
Sep 16 19:38:22.603913 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.603893    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 16 19:38:22.604015 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.603894    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 16 19:38:22.605108 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.605090    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-7nrvt\""
Sep 16 19:38:22.605239 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.605213    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Sep 16 19:38:22.610417 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.610389    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"]
Sep 16 19:38:22.633448 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.633406    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqnb4\" (UniqueName: \"kubernetes.io/projected/5c4543f5-20e7-4501-80aa-f6bca6fe67bc-kube-api-access-kqnb4\") pod \"e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c\" (UID: \"5c4543f5-20e7-4501-80aa-f6bca6fe67bc\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"
Sep 16 19:38:22.633662 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.633474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4543f5-20e7-4501-80aa-f6bca6fe67bc-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c\" (UID: \"5c4543f5-20e7-4501-80aa-f6bca6fe67bc\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"
Sep 16 19:38:22.694461 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.694422    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:22.734838 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.734795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqnb4\" (UniqueName: \"kubernetes.io/projected/5c4543f5-20e7-4501-80aa-f6bca6fe67bc-kube-api-access-kqnb4\") pod \"e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c\" (UID: \"5c4543f5-20e7-4501-80aa-f6bca6fe67bc\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"
Sep 16 19:38:22.735032 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.734891    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4543f5-20e7-4501-80aa-f6bca6fe67bc-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c\" (UID: \"5c4543f5-20e7-4501-80aa-f6bca6fe67bc\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"
Sep 16 19:38:22.735340 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.735320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4543f5-20e7-4501-80aa-f6bca6fe67bc-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c\" (UID: \"5c4543f5-20e7-4501-80aa-f6bca6fe67bc\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"
Sep 16 19:38:22.744430 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.744400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqnb4\" (UniqueName: \"kubernetes.io/projected/5c4543f5-20e7-4501-80aa-f6bca6fe67bc-kube-api-access-kqnb4\") pod \"e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c\" (UID: \"5c4543f5-20e7-4501-80aa-f6bca6fe67bc\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"
Sep 16 19:38:22.912394 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.912292    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"
Sep 16 19:38:22.929076 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.929041    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"]
Sep 16 19:38:22.932219 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.932191    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"
Sep 16 19:38:22.935222 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.935186    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-kserve-self-signed-certs\""
Sep 16 19:38:22.935371 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.935186    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-9t8bn\""
Sep 16 19:38:22.944459 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.944376    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"]
Sep 16 19:38:22.975454 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:22.975417    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:23.038486 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:23.038441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5\" (UID: \"9c6fc38c-b825-42e4-9dce-f38ec800a5d0\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"
Sep 16 19:38:23.038733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:23.038516    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw9gw\" (UniqueName: \"kubernetes.io/projected/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-kube-api-access-fw9gw\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5\" (UID: \"9c6fc38c-b825-42e4-9dce-f38ec800a5d0\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"
Sep 16 19:38:23.139892 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:23.139851    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5\" (UID: \"9c6fc38c-b825-42e4-9dce-f38ec800a5d0\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"
Sep 16 19:38:23.140087 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:23.139899    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw9gw\" (UniqueName: \"kubernetes.io/projected/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-kube-api-access-fw9gw\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5\" (UID: \"9c6fc38c-b825-42e4-9dce-f38ec800a5d0\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"
Sep 16 19:38:23.142785 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:23.142760    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5\" (UID: \"9c6fc38c-b825-42e4-9dce-f38ec800a5d0\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"
Sep 16 19:38:23.153428 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:23.153399    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw9gw\" (UniqueName: \"kubernetes.io/projected/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-kube-api-access-fw9gw\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5\" (UID: \"9c6fc38c-b825-42e4-9dce-f38ec800a5d0\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"
Sep 16 19:38:23.197409 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:23.197283    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"]
Sep 16 19:38:23.200519 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:38:23.200487    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4543f5_20e7_4501_80aa_f6bca6fe67bc.slice/crio-eaa026e3a4a8659ab9da85806533ebd08016c3b871b35b33843e525136053990 WatchSource:0}: Error finding container eaa026e3a4a8659ab9da85806533ebd08016c3b871b35b33843e525136053990: Status 404 returned error can't find the container with id eaa026e3a4a8659ab9da85806533ebd08016c3b871b35b33843e525136053990
Sep 16 19:38:23.255848 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:23.255800    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"
Sep 16 19:38:23.537454 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:23.537418    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"]
Sep 16 19:38:23.539530 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:38:23.539501    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6fc38c_b825_42e4_9dce_f38ec800a5d0.slice/crio-c09caacff067cccea450d2f8063411a651a65b1d35f5bdf5b9be14a9689a1fa8 WatchSource:0}: Error finding container c09caacff067cccea450d2f8063411a651a65b1d35f5bdf5b9be14a9689a1fa8: Status 404 returned error can't find the container with id c09caacff067cccea450d2f8063411a651a65b1d35f5bdf5b9be14a9689a1fa8
Sep 16 19:38:23.576815 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:23.576778    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" event={"ID":"9c6fc38c-b825-42e4-9dce-f38ec800a5d0","Type":"ContainerStarted","Data":"c09caacff067cccea450d2f8063411a651a65b1d35f5bdf5b9be14a9689a1fa8"}
Sep 16 19:38:23.578601 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:23.578551    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c" event={"ID":"5c4543f5-20e7-4501-80aa-f6bca6fe67bc","Type":"ContainerStarted","Data":"0b1fe5d730b010a75d9849c9ae226745f877a4d9665a2e1fff20acfa2169541a"}
Sep 16 19:38:23.578734 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:23.578618    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c" event={"ID":"5c4543f5-20e7-4501-80aa-f6bca6fe67bc","Type":"ContainerStarted","Data":"eaa026e3a4a8659ab9da85806533ebd08016c3b871b35b33843e525136053990"}
Sep 16 19:38:24.206054 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:24.206010    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"]
Sep 16 19:38:24.211959 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:24.211929    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:24.212125 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:24.211984    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"]
Sep 16 19:38:24.272130 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:24.272090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:24.588211 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:24.588162    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" event={"ID":"9c6fc38c-b825-42e4-9dce-f38ec800a5d0","Type":"ContainerStarted","Data":"5ee9840c2c4a244402cd6cd912f7ed9253581933837d0e8637afffc0c2f58771"}
Sep 16 19:38:24.588453 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:24.588369    2582 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" secret="" err="secret \"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-9t8bn\" not found"
Sep 16 19:38:24.588560 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:24.588448    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"
Sep 16 19:38:24.605930 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:24.605867    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" podStartSLOduration=2.605843823 podStartE2EDuration="2.605843823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:38:24.604771097 +0000 UTC m=+3162.435379729" watchObservedRunningTime="2026-09-16 19:38:24.605843823 +0000 UTC m=+3162.436452645"
Sep 16 19:38:24.655872 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:24.655822    2582 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:24.656106 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:24.655923    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs podName:9c6fc38c-b825-42e4-9dce-f38ec800a5d0 nodeName:}" failed. No retries permitted until 2026-09-16 19:38:25.15589764 +0000 UTC m=+3162.986506183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" (UID: "9c6fc38c-b825-42e4-9dce-f38ec800a5d0") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:25.160873 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:25.160815    2582 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:25.161154 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:25.160925    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs podName:9c6fc38c-b825-42e4-9dce-f38ec800a5d0 nodeName:}" failed. No retries permitted until 2026-09-16 19:38:26.160902757 +0000 UTC m=+3163.991511280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" (UID: "9c6fc38c-b825-42e4-9dce-f38ec800a5d0") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:25.595365 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:25.595293    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c" podUID="5c4543f5-20e7-4501-80aa-f6bca6fe67bc" containerName="storage-initializer" containerID="cri-o://0b1fe5d730b010a75d9849c9ae226745f877a4d9665a2e1fff20acfa2169541a" gracePeriod=30
Sep 16 19:38:25.711158 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:25.711113    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:25.832897 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:25.832859    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:26.172257 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:26.172202    2582 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:26.172531 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:26.172305    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs podName:9c6fc38c-b825-42e4-9dce-f38ec800a5d0 nodeName:}" failed. No retries permitted until 2026-09-16 19:38:28.172288509 +0000 UTC m=+3166.002897032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" (UID: "9c6fc38c-b825-42e4-9dce-f38ec800a5d0") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:28.192492 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:28.192452    2582 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:28.192905 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:28.192525    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs podName:9c6fc38c-b825-42e4-9dce-f38ec800a5d0 nodeName:}" failed. No retries permitted until 2026-09-16 19:38:32.192508628 +0000 UTC m=+3170.023117151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" (UID: "9c6fc38c-b825-42e4-9dce-f38ec800a5d0") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:28.546302 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.546274    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"
Sep 16 19:38:28.596390 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.596352    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4543f5-20e7-4501-80aa-f6bca6fe67bc-kserve-provision-location\") pod \"5c4543f5-20e7-4501-80aa-f6bca6fe67bc\" (UID: \"5c4543f5-20e7-4501-80aa-f6bca6fe67bc\") "
Sep 16 19:38:28.596626 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.596440    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqnb4\" (UniqueName: \"kubernetes.io/projected/5c4543f5-20e7-4501-80aa-f6bca6fe67bc-kube-api-access-kqnb4\") pod \"5c4543f5-20e7-4501-80aa-f6bca6fe67bc\" (UID: \"5c4543f5-20e7-4501-80aa-f6bca6fe67bc\") "
Sep 16 19:38:28.598950 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.598911    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4543f5-20e7-4501-80aa-f6bca6fe67bc-kube-api-access-kqnb4" (OuterVolumeSpecName: "kube-api-access-kqnb4") pod "5c4543f5-20e7-4501-80aa-f6bca6fe67bc" (UID: "5c4543f5-20e7-4501-80aa-f6bca6fe67bc"). InnerVolumeSpecName "kube-api-access-kqnb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:28.614565 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.614529    2582 generic.go:356] "Generic (PLEG): container finished" podID="5c4543f5-20e7-4501-80aa-f6bca6fe67bc" containerID="0b1fe5d730b010a75d9849c9ae226745f877a4d9665a2e1fff20acfa2169541a" exitCode=0
Sep 16 19:38:28.614717 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.614597    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"
Sep 16 19:38:28.614717 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.614608    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c" event={"ID":"5c4543f5-20e7-4501-80aa-f6bca6fe67bc","Type":"ContainerDied","Data":"0b1fe5d730b010a75d9849c9ae226745f877a4d9665a2e1fff20acfa2169541a"}
Sep 16 19:38:28.614717 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.614669    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c" event={"ID":"5c4543f5-20e7-4501-80aa-f6bca6fe67bc","Type":"ContainerDied","Data":"eaa026e3a4a8659ab9da85806533ebd08016c3b871b35b33843e525136053990"}
Sep 16 19:38:28.614717 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.614694    2582 scope.go:117] "RemoveContainer" containerID="0b1fe5d730b010a75d9849c9ae226745f877a4d9665a2e1fff20acfa2169541a"
Sep 16 19:38:28.623667 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.623647    2582 scope.go:117] "RemoveContainer" containerID="0b1fe5d730b010a75d9849c9ae226745f877a4d9665a2e1fff20acfa2169541a"
Sep 16 19:38:28.623967 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:28.623943    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1fe5d730b010a75d9849c9ae226745f877a4d9665a2e1fff20acfa2169541a\": container with ID starting with 0b1fe5d730b010a75d9849c9ae226745f877a4d9665a2e1fff20acfa2169541a not found: ID does not exist" containerID="0b1fe5d730b010a75d9849c9ae226745f877a4d9665a2e1fff20acfa2169541a"
Sep 16 19:38:28.624076 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.623976    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1fe5d730b010a75d9849c9ae226745f877a4d9665a2e1fff20acfa2169541a"} err="failed to get container status \"0b1fe5d730b010a75d9849c9ae226745f877a4d9665a2e1fff20acfa2169541a\": rpc error: code = NotFound desc = could not find container \"0b1fe5d730b010a75d9849c9ae226745f877a4d9665a2e1fff20acfa2169541a\": container with ID starting with 0b1fe5d730b010a75d9849c9ae226745f877a4d9665a2e1fff20acfa2169541a not found: ID does not exist"
Sep 16 19:38:28.672407 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.672349    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4543f5-20e7-4501-80aa-f6bca6fe67bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c4543f5-20e7-4501-80aa-f6bca6fe67bc" (UID: "5c4543f5-20e7-4501-80aa-f6bca6fe67bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:28.697422 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.697384    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4543f5-20e7-4501-80aa-f6bca6fe67bc-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:38:28.697422 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.697417    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqnb4\" (UniqueName: \"kubernetes.io/projected/5c4543f5-20e7-4501-80aa-f6bca6fe67bc-kube-api-access-kqnb4\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:38:28.947226 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.947169    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"]
Sep 16 19:38:28.948953 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:28.948919    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-8dc755469-rhs6c"]
Sep 16 19:38:30.704391 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:30.704353    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c4543f5-20e7-4501-80aa-f6bca6fe67bc" path="/var/lib/kubelet/pods/5c4543f5-20e7-4501-80aa-f6bca6fe67bc/volumes"
Sep 16 19:38:30.763401 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:30.763368    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:31.515869 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:31.515827    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds" podUID="391953c9-797b-47a3-9d49-b842fa3197f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:32.230128 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:32.230080    2582 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:32.230604 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:32.230170    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs podName:9c6fc38c-b825-42e4-9dce-f38ec800a5d0 nodeName:}" failed. No retries permitted until 2026-09-16 19:38:40.230150224 +0000 UTC m=+3178.060758768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" (UID: "9c6fc38c-b825-42e4-9dce-f38ec800a5d0") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:35.448118 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:35.448077    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:35.656106 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:35.656055    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:40.311943 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:40.311897    2582 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:40.312480 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:40.311998    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs podName:9c6fc38c-b825-42e4-9dce-f38ec800a5d0 nodeName:}" failed. No retries permitted until 2026-09-16 19:38:56.311982997 +0000 UTC m=+3194.142591519 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" (UID: "9c6fc38c-b825-42e4-9dce-f38ec800a5d0") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:40.617814 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:40.617695    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" podUID="9c6fc38c-b825-42e4-9dce-f38ec800a5d0" containerName="main" containerID="cri-o://5ee9840c2c4a244402cd6cd912f7ed9253581933837d0e8637afffc0c2f58771" gracePeriod=45
Sep 16 19:38:41.515108 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:41.515066    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds" podUID="391953c9-797b-47a3-9d49-b842fa3197f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:45.110074 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:45.110033    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:46.711822 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:46.711766    2582 generic.go:356] "Generic (PLEG): container finished" podID="391953c9-797b-47a3-9d49-b842fa3197f8" containerID="5e84793e8a33d3772b6b6a7a248b3cdf25e5c547b3391a086b87640810be3cd5" exitCode=0
Sep 16 19:38:46.712728 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:46.712701    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds" event={"ID":"391953c9-797b-47a3-9d49-b842fa3197f8","Type":"ContainerDied","Data":"5e84793e8a33d3772b6b6a7a248b3cdf25e5c547b3391a086b87640810be3cd5"}
Sep 16 19:38:46.797119 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:46.797050    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"
Sep 16 19:38:46.976875 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:46.976814    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwqsn\" (UniqueName: \"kubernetes.io/projected/391953c9-797b-47a3-9d49-b842fa3197f8-kube-api-access-wwqsn\") pod \"391953c9-797b-47a3-9d49-b842fa3197f8\" (UID: \"391953c9-797b-47a3-9d49-b842fa3197f8\") "
Sep 16 19:38:46.976875 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:46.976886    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/391953c9-797b-47a3-9d49-b842fa3197f8-tls-certs\") pod \"391953c9-797b-47a3-9d49-b842fa3197f8\" (UID: \"391953c9-797b-47a3-9d49-b842fa3197f8\") "
Sep 16 19:38:46.979390 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:46.979356    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391953c9-797b-47a3-9d49-b842fa3197f8-kube-api-access-wwqsn" (OuterVolumeSpecName: "kube-api-access-wwqsn") pod "391953c9-797b-47a3-9d49-b842fa3197f8" (UID: "391953c9-797b-47a3-9d49-b842fa3197f8"). InnerVolumeSpecName "kube-api-access-wwqsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:46.979498 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:46.979358    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391953c9-797b-47a3-9d49-b842fa3197f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "391953c9-797b-47a3-9d49-b842fa3197f8" (UID: "391953c9-797b-47a3-9d49-b842fa3197f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:47.078171 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:47.078056    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwqsn\" (UniqueName: \"kubernetes.io/projected/391953c9-797b-47a3-9d49-b842fa3197f8-kube-api-access-wwqsn\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:38:47.078171 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:47.078105    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/391953c9-797b-47a3-9d49-b842fa3197f8-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:38:47.717896 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:47.717854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds" event={"ID":"391953c9-797b-47a3-9d49-b842fa3197f8","Type":"ContainerDied","Data":"e1f17ed0438af39e475a547cc95e92d428dcecb335a944b9834459238d70c8ac"}
Sep 16 19:38:47.718382 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:47.717921    2582 scope.go:117] "RemoveContainer" containerID="5e84793e8a33d3772b6b6a7a248b3cdf25e5c547b3391a086b87640810be3cd5"
Sep 16 19:38:47.718382 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:47.717873    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"
Sep 16 19:38:47.740109 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:47.740066    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"]
Sep 16 19:38:47.743560 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:47.743527    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-9xdds"]
Sep 16 19:38:48.704266 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:48.704221    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="391953c9-797b-47a3-9d49-b842fa3197f8" path="/var/lib/kubelet/pods/391953c9-797b-47a3-9d49-b842fa3197f8/volumes"
Sep 16 19:38:55.598741 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:55.598697    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" podUID="9c6fc38c-b825-42e4-9dce-f38ec800a5d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:56.352973 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:56.352934    2582 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:56.353158 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:38:56.353023    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs podName:9c6fc38c-b825-42e4-9dce-f38ec800a5d0 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:28.353007814 +0000 UTC m=+3226.183616336 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" (UID: "9c6fc38c-b825-42e4-9dce-f38ec800a5d0") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:59.557844 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:59.557765    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:59.654026 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:38:59.653988    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:02.972620 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:02.972556    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:03.074029 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:03.073976    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:05.598076 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.598021    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" podUID="9c6fc38c-b825-42e4-9dce-f38ec800a5d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:39:05.607138 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.607101    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"]
Sep 16 19:39:05.607531 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.607517    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4543f5-20e7-4501-80aa-f6bca6fe67bc" containerName="storage-initializer"
Sep 16 19:39:05.607596 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.607533    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4543f5-20e7-4501-80aa-f6bca6fe67bc" containerName="storage-initializer"
Sep 16 19:39:05.607596 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.607553    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="391953c9-797b-47a3-9d49-b842fa3197f8" containerName="main"
Sep 16 19:39:05.607596 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.607558    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="391953c9-797b-47a3-9d49-b842fa3197f8" containerName="main"
Sep 16 19:39:05.607702 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.607630    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c4543f5-20e7-4501-80aa-f6bca6fe67bc" containerName="storage-initializer"
Sep 16 19:39:05.607702 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.607643    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="391953c9-797b-47a3-9d49-b842fa3197f8" containerName="main"
Sep 16 19:39:05.610012 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.609991    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.615485 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.615457    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-p9thc\""
Sep 16 19:39:05.615706 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.615492    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 19:39:05.615831 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.615812    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 19:39:05.615910 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.615891    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 19:39:05.631728 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.631690    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"]
Sep 16 19:39:05.639048 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.638997    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clchw\" (UniqueName: \"kubernetes.io/projected/361ee333-832f-445d-937a-b3a5d18f53a2-kube-api-access-clchw\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.639048 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.639045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.639348 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.639178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.639348 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.639228    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.639348 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.639329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.639508 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.639372    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/361ee333-832f-445d-937a-b3a5d18f53a2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.639508 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.639435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.642783 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.642734    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:05.740723 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.740659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.740723 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.740725    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.741040 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.740761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.741040 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.740784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/361ee333-832f-445d-937a-b3a5d18f53a2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.741040 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.740819    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.741040 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.740875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clchw\" (UniqueName: \"kubernetes.io/projected/361ee333-832f-445d-937a-b3a5d18f53a2-kube-api-access-clchw\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.741040 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.740893    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.741294 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.741200    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.741294 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.741216    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.741294 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.741241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.741426 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.741326    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.743748 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.743719    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.743979 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.743957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/361ee333-832f-445d-937a-b3a5d18f53a2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.764188 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.764149    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clchw\" (UniqueName: \"kubernetes.io/projected/361ee333-832f-445d-937a-b3a5d18f53a2-kube-api-access-clchw\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:05.923462 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:05.923354    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:06.020807 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:06.020749    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:06.144946 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:06.144880    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:06.210259 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:06.210223    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"]
Sep 16 19:39:06.253136 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:06.253111    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:39:06.810475 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:06.810424    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" event={"ID":"361ee333-832f-445d-937a-b3a5d18f53a2","Type":"ContainerStarted","Data":"5f15ce2ff763024be0d2b37d762e5131044461aeda96ed973623f2943e82ea4d"}
Sep 16 19:39:11.819967 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:11.819915    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:11.919970 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:11.919915    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:12.122028 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.122000    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"
Sep 16 19:39:12.210267 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.210212    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw9gw\" (UniqueName: \"kubernetes.io/projected/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-kube-api-access-fw9gw\") pod \"9c6fc38c-b825-42e4-9dce-f38ec800a5d0\" (UID: \"9c6fc38c-b825-42e4-9dce-f38ec800a5d0\") "
Sep 16 19:39:12.210530 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.210431    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs\") pod \"9c6fc38c-b825-42e4-9dce-f38ec800a5d0\" (UID: \"9c6fc38c-b825-42e4-9dce-f38ec800a5d0\") "
Sep 16 19:39:12.213000 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.212972    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c6fc38c-b825-42e4-9dce-f38ec800a5d0" (UID: "9c6fc38c-b825-42e4-9dce-f38ec800a5d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:39:12.213101 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.212978    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-kube-api-access-fw9gw" (OuterVolumeSpecName: "kube-api-access-fw9gw") pod "9c6fc38c-b825-42e4-9dce-f38ec800a5d0" (UID: "9c6fc38c-b825-42e4-9dce-f38ec800a5d0"). InnerVolumeSpecName "kube-api-access-fw9gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:39:12.311850 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.311802    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:39:12.311850 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.311842    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw9gw\" (UniqueName: \"kubernetes.io/projected/9c6fc38c-b825-42e4-9dce-f38ec800a5d0-kube-api-access-fw9gw\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:39:12.846776 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.846714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" event={"ID":"361ee333-832f-445d-937a-b3a5d18f53a2","Type":"ContainerStarted","Data":"8b60c4e8bb082969d08d87880035942f6129cff4bc1a7a07c7b76083b54e0f0b"}
Sep 16 19:39:12.848259 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.848219    2582 generic.go:356] "Generic (PLEG): container finished" podID="9c6fc38c-b825-42e4-9dce-f38ec800a5d0" containerID="5ee9840c2c4a244402cd6cd912f7ed9253581933837d0e8637afffc0c2f58771" exitCode=0
Sep 16 19:39:12.848417 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.848278    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" event={"ID":"9c6fc38c-b825-42e4-9dce-f38ec800a5d0","Type":"ContainerDied","Data":"5ee9840c2c4a244402cd6cd912f7ed9253581933837d0e8637afffc0c2f58771"}
Sep 16 19:39:12.848417 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.848320    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5" event={"ID":"9c6fc38c-b825-42e4-9dce-f38ec800a5d0","Type":"ContainerDied","Data":"c09caacff067cccea450d2f8063411a651a65b1d35f5bdf5b9be14a9689a1fa8"}
Sep 16 19:39:12.848417 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.848345    2582 scope.go:117] "RemoveContainer" containerID="5ee9840c2c4a244402cd6cd912f7ed9253581933837d0e8637afffc0c2f58771"
Sep 16 19:39:12.848417 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.848284    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"
Sep 16 19:39:12.860827 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.860757    2582 scope.go:117] "RemoveContainer" containerID="5ee9840c2c4a244402cd6cd912f7ed9253581933837d0e8637afffc0c2f58771"
Sep 16 19:39:12.861484 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:39:12.861453    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee9840c2c4a244402cd6cd912f7ed9253581933837d0e8637afffc0c2f58771\": container with ID starting with 5ee9840c2c4a244402cd6cd912f7ed9253581933837d0e8637afffc0c2f58771 not found: ID does not exist" containerID="5ee9840c2c4a244402cd6cd912f7ed9253581933837d0e8637afffc0c2f58771"
Sep 16 19:39:12.861673 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.861490    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee9840c2c4a244402cd6cd912f7ed9253581933837d0e8637afffc0c2f58771"} err="failed to get container status \"5ee9840c2c4a244402cd6cd912f7ed9253581933837d0e8637afffc0c2f58771\": rpc error: code = NotFound desc = could not find container \"5ee9840c2c4a244402cd6cd912f7ed9253581933837d0e8637afffc0c2f58771\": container with ID starting with 5ee9840c2c4a244402cd6cd912f7ed9253581933837d0e8637afffc0c2f58771 not found: ID does not exist"
Sep 16 19:39:12.888181 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.888128    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"]
Sep 16 19:39:12.889828 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:12.889800    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddkwzq5"]
Sep 16 19:39:13.855523 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:13.855476    2582 generic.go:356] "Generic (PLEG): container finished" podID="361ee333-832f-445d-937a-b3a5d18f53a2" containerID="8b60c4e8bb082969d08d87880035942f6129cff4bc1a7a07c7b76083b54e0f0b" exitCode=0
Sep 16 19:39:13.856144 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:13.855561    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" event={"ID":"361ee333-832f-445d-937a-b3a5d18f53a2","Type":"ContainerDied","Data":"8b60c4e8bb082969d08d87880035942f6129cff4bc1a7a07c7b76083b54e0f0b"}
Sep 16 19:39:14.508009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:14.507895    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:14.706007 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:14.705961    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c6fc38c-b825-42e4-9dce-f38ec800a5d0" path="/var/lib/kubelet/pods/9c6fc38c-b825-42e4-9dce-f38ec800a5d0/volumes"
Sep 16 19:39:14.865121 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:14.865076    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" event={"ID":"361ee333-832f-445d-937a-b3a5d18f53a2","Type":"ContainerStarted","Data":"6273f3e3ed3d18d9bc535ac08e40b2cd286f7b917038eb8b7c227b7190be492a"}
Sep 16 19:39:14.865610 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:14.865145    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" event={"ID":"361ee333-832f-445d-937a-b3a5d18f53a2","Type":"ContainerStarted","Data":"0ec8b9365bd720534aba17ca6ae89cc98330ff405ea44635ceec24713003c5d7"}
Sep 16 19:39:14.893076 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:14.892995    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podStartSLOduration=2.584594209 podStartE2EDuration="9.89296449s" totalImagesPullingTime="6.023161558s" totalInitContainerRuntime="1.285208723s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:14.891041658 +0000 UTC m=+3212.721650216" watchObservedRunningTime="2026-09-16 19:39:14.89296449 +0000 UTC m=+3212.723573034"
Sep 16 19:39:14.924036 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:14.923988    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:15.473301 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:15.473242    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:15.923800 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:15.923739    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:39:15.926030 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:15.925982    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:39:25.924452 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:25.924399    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:39:35.924941 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:35.924800    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:39:39.083794 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:39.083752    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:45.924816 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:45.924766    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:39:49.151734 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:49.151686    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:55.924632 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:39:55.924549    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:40:05.924339 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:40:05.924268    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:40:15.924803 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:40:15.924735    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:40:25.924912 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:40:25.924844    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:40:35.924486 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:40:35.924426    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:40:38.451780 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:40:38.451733    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:38.552290 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:40:38.552246    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:45.924294 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:40:45.924235    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:40:55.924126 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:40:55.924070    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:41:03.871486 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:03.871399    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:04.088878 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.088832    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"]
Sep 16 19:41:04.089231 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.089212    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c6fc38c-b825-42e4-9dce-f38ec800a5d0" containerName="main"
Sep 16 19:41:04.089231 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.089228    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c6fc38c-b825-42e4-9dce-f38ec800a5d0" containerName="main"
Sep 16 19:41:04.089418 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.089304    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c6fc38c-b825-42e4-9dce-f38ec800a5d0" containerName="main"
Sep 16 19:41:04.092874 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.092851    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"
Sep 16 19:41:04.097445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.097420    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 19:41:04.097590 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.097501    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 19:41:04.097590 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.097535    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 19:41:04.097590 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.097535    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-j6zng\""
Sep 16 19:41:04.097819 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.097536    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-dc5sv\""
Sep 16 19:41:04.104538 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.104511    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"]
Sep 16 19:41:04.181105 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.181028    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rmm2\" (UniqueName: \"kubernetes.io/projected/d5d295a0-49a1-4118-a10d-d20e9824d24e-kube-api-access-9rmm2\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff\" (UID: \"d5d295a0-49a1-4118-a10d-d20e9824d24e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"
Sep 16 19:41:04.181105 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.181101    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d295a0-49a1-4118-a10d-d20e9824d24e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff\" (UID: \"d5d295a0-49a1-4118-a10d-d20e9824d24e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"
Sep 16 19:41:04.211567 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.211528    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:04.281713 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.281679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rmm2\" (UniqueName: \"kubernetes.io/projected/d5d295a0-49a1-4118-a10d-d20e9824d24e-kube-api-access-9rmm2\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff\" (UID: \"d5d295a0-49a1-4118-a10d-d20e9824d24e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"
Sep 16 19:41:04.281887 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.281747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d295a0-49a1-4118-a10d-d20e9824d24e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff\" (UID: \"d5d295a0-49a1-4118-a10d-d20e9824d24e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"
Sep 16 19:41:04.284443 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.284414    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d295a0-49a1-4118-a10d-d20e9824d24e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff\" (UID: \"d5d295a0-49a1-4118-a10d-d20e9824d24e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"
Sep 16 19:41:04.292199 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.292172    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rmm2\" (UniqueName: \"kubernetes.io/projected/d5d295a0-49a1-4118-a10d-d20e9824d24e-kube-api-access-9rmm2\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff\" (UID: \"d5d295a0-49a1-4118-a10d-d20e9824d24e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"
Sep 16 19:41:04.403589 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.403540    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"
Sep 16 19:41:04.678877 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:04.678841    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"]
Sep 16 19:41:04.682962 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:41:04.682933    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d295a0_49a1_4118_a10d_d20e9824d24e.slice/crio-58a172f23f32e4c5ac8dacaf6c1d343e36e92597efbbe1a8f5c5b3f7fa559901 WatchSource:0}: Error finding container 58a172f23f32e4c5ac8dacaf6c1d343e36e92597efbbe1a8f5c5b3f7fa559901: Status 404 returned error can't find the container with id 58a172f23f32e4c5ac8dacaf6c1d343e36e92597efbbe1a8f5c5b3f7fa559901
Sep 16 19:41:05.469445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:05.469407    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff" event={"ID":"d5d295a0-49a1-4118-a10d-d20e9824d24e","Type":"ContainerStarted","Data":"98930fc1f8418a9b3766c2967e028394aed7ca32d3c84672afc173ef59b6fd2d"}
Sep 16 19:41:05.469893 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:05.469460    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff" event={"ID":"d5d295a0-49a1-4118-a10d-d20e9824d24e","Type":"ContainerStarted","Data":"58a172f23f32e4c5ac8dacaf6c1d343e36e92597efbbe1a8f5c5b3f7fa559901"}
Sep 16 19:41:05.469893 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:05.469562    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"
Sep 16 19:41:05.488824 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:05.488741    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff" podStartSLOduration=1.488720054 podStartE2EDuration="1.488720054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:05.487640541 +0000 UTC m=+3323.318249086" watchObservedRunningTime="2026-09-16 19:41:05.488720054 +0000 UTC m=+3323.319328600"
Sep 16 19:41:05.924745 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:05.924694    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:41:10.231168 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:10.231126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:15.924913 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:15.924859    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:41:25.934335 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:25.934303    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:41:25.942158 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:25.942132    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:41:31.446678 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:31.446637    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"]
Sep 16 19:41:31.447268 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:31.447020    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="modelcar" containerID="cri-o://6273f3e3ed3d18d9bc535ac08e40b2cd286f7b917038eb8b7c227b7190be492a" gracePeriod=60
Sep 16 19:41:31.450938 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:31.450904    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:31.519771 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:31.519738    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:31.606495 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:31.606465    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd_361ee333-832f-445d-937a-b3a5d18f53a2/modelcar/0.log"
Sep 16 19:41:31.607201 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:31.607170    2582 generic.go:356] "Generic (PLEG): container finished" podID="361ee333-832f-445d-937a-b3a5d18f53a2" containerID="6273f3e3ed3d18d9bc535ac08e40b2cd286f7b917038eb8b7c227b7190be492a" exitCode=143
Sep 16 19:41:31.607328 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:31.607252    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" event={"ID":"361ee333-832f-445d-937a-b3a5d18f53a2","Type":"ContainerDied","Data":"6273f3e3ed3d18d9bc535ac08e40b2cd286f7b917038eb8b7c227b7190be492a"}
Sep 16 19:41:36.476765 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:36.476725    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"
Sep 16 19:41:46.470975 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:46.470930    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" containerID="cri-o://0ec8b9365bd720534aba17ca6ae89cc98330ff405ea44635ceec24713003c5d7" gracePeriod=45
Sep 16 19:41:46.682795 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:46.682758    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:46.750913 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:46.750803    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:46.936017 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:46.935972    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 16 19:41:47.329474 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.329451    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd_361ee333-832f-445d-937a-b3a5d18f53a2/modelcar/0.log"
Sep 16 19:41:47.332436 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.332413    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:41:47.376725 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.376687    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-home\") pod \"361ee333-832f-445d-937a-b3a5d18f53a2\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") "
Sep 16 19:41:47.376725 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.376730    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-kserve-provision-location\") pod \"361ee333-832f-445d-937a-b3a5d18f53a2\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") "
Sep 16 19:41:47.376942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.376760    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-tmp-dir\") pod \"361ee333-832f-445d-937a-b3a5d18f53a2\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") "
Sep 16 19:41:47.376942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.376804    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/361ee333-832f-445d-937a-b3a5d18f53a2-tls-certs\") pod \"361ee333-832f-445d-937a-b3a5d18f53a2\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") "
Sep 16 19:41:47.376942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.376824    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-dshm\") pod \"361ee333-832f-445d-937a-b3a5d18f53a2\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") "
Sep 16 19:41:47.376942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.376894    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clchw\" (UniqueName: \"kubernetes.io/projected/361ee333-832f-445d-937a-b3a5d18f53a2-kube-api-access-clchw\") pod \"361ee333-832f-445d-937a-b3a5d18f53a2\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") "
Sep 16 19:41:47.376942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.376936    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-model-cache\") pod \"361ee333-832f-445d-937a-b3a5d18f53a2\" (UID: \"361ee333-832f-445d-937a-b3a5d18f53a2\") "
Sep 16 19:41:47.377213 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.377057    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "361ee333-832f-445d-937a-b3a5d18f53a2" (UID: "361ee333-832f-445d-937a-b3a5d18f53a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:47.377307 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.377284    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:41:47.377367 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.377295    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "361ee333-832f-445d-937a-b3a5d18f53a2" (UID: "361ee333-832f-445d-937a-b3a5d18f53a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:47.377484 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.377459    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-home" (OuterVolumeSpecName: "home") pod "361ee333-832f-445d-937a-b3a5d18f53a2" (UID: "361ee333-832f-445d-937a-b3a5d18f53a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:47.379362 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.379340    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-dshm" (OuterVolumeSpecName: "dshm") pod "361ee333-832f-445d-937a-b3a5d18f53a2" (UID: "361ee333-832f-445d-937a-b3a5d18f53a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:47.379468 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.379378    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361ee333-832f-445d-937a-b3a5d18f53a2-kube-api-access-clchw" (OuterVolumeSpecName: "kube-api-access-clchw") pod "361ee333-832f-445d-937a-b3a5d18f53a2" (UID: "361ee333-832f-445d-937a-b3a5d18f53a2"). InnerVolumeSpecName "kube-api-access-clchw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:41:47.379674 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.379657    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/361ee333-832f-445d-937a-b3a5d18f53a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "361ee333-832f-445d-937a-b3a5d18f53a2" (UID: "361ee333-832f-445d-937a-b3a5d18f53a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:41:47.388800 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.388774    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "361ee333-832f-445d-937a-b3a5d18f53a2" (UID: "361ee333-832f-445d-937a-b3a5d18f53a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:41:47.478392 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.478354    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:41:47.478392 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.478387    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/361ee333-832f-445d-937a-b3a5d18f53a2-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:41:47.478392 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.478395    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:41:47.478865 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.478407    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clchw\" (UniqueName: \"kubernetes.io/projected/361ee333-832f-445d-937a-b3a5d18f53a2-kube-api-access-clchw\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:41:47.478865 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.478418    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:41:47.478865 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.478425    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/361ee333-832f-445d-937a-b3a5d18f53a2-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:41:47.682539 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.682454    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd_361ee333-832f-445d-937a-b3a5d18f53a2/modelcar/0.log"
Sep 16 19:41:47.683139 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.683108    2582 generic.go:356] "Generic (PLEG): container finished" podID="361ee333-832f-445d-937a-b3a5d18f53a2" containerID="0ec8b9365bd720534aba17ca6ae89cc98330ff405ea44635ceec24713003c5d7" exitCode=0
Sep 16 19:41:47.683263 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.683177    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"
Sep 16 19:41:47.683263 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.683191    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" event={"ID":"361ee333-832f-445d-937a-b3a5d18f53a2","Type":"ContainerDied","Data":"0ec8b9365bd720534aba17ca6ae89cc98330ff405ea44635ceec24713003c5d7"}
Sep 16 19:41:47.683394 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.683271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd" event={"ID":"361ee333-832f-445d-937a-b3a5d18f53a2","Type":"ContainerDied","Data":"5f15ce2ff763024be0d2b37d762e5131044461aeda96ed973623f2943e82ea4d"}
Sep 16 19:41:47.683394 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.683320    2582 scope.go:117] "RemoveContainer" containerID="6273f3e3ed3d18d9bc535ac08e40b2cd286f7b917038eb8b7c227b7190be492a"
Sep 16 19:41:47.692501 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.692483    2582 scope.go:117] "RemoveContainer" containerID="0ec8b9365bd720534aba17ca6ae89cc98330ff405ea44635ceec24713003c5d7"
Sep 16 19:41:47.700421 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.700399    2582 scope.go:117] "RemoveContainer" containerID="8b60c4e8bb082969d08d87880035942f6129cff4bc1a7a07c7b76083b54e0f0b"
Sep 16 19:41:47.708611 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.707566    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"]
Sep 16 19:41:47.710680 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.710638    2582 scope.go:117] "RemoveContainer" containerID="6273f3e3ed3d18d9bc535ac08e40b2cd286f7b917038eb8b7c227b7190be492a"
Sep 16 19:41:47.711164 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:41:47.711130    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6273f3e3ed3d18d9bc535ac08e40b2cd286f7b917038eb8b7c227b7190be492a\": container with ID starting with 6273f3e3ed3d18d9bc535ac08e40b2cd286f7b917038eb8b7c227b7190be492a not found: ID does not exist" containerID="6273f3e3ed3d18d9bc535ac08e40b2cd286f7b917038eb8b7c227b7190be492a"
Sep 16 19:41:47.711254 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.711172    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6273f3e3ed3d18d9bc535ac08e40b2cd286f7b917038eb8b7c227b7190be492a"} err="failed to get container status \"6273f3e3ed3d18d9bc535ac08e40b2cd286f7b917038eb8b7c227b7190be492a\": rpc error: code = NotFound desc = could not find container \"6273f3e3ed3d18d9bc535ac08e40b2cd286f7b917038eb8b7c227b7190be492a\": container with ID starting with 6273f3e3ed3d18d9bc535ac08e40b2cd286f7b917038eb8b7c227b7190be492a not found: ID does not exist"
Sep 16 19:41:47.711254 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.711199    2582 scope.go:117] "RemoveContainer" containerID="0ec8b9365bd720534aba17ca6ae89cc98330ff405ea44635ceec24713003c5d7"
Sep 16 19:41:47.711640 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:41:47.711617    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec8b9365bd720534aba17ca6ae89cc98330ff405ea44635ceec24713003c5d7\": container with ID starting with 0ec8b9365bd720534aba17ca6ae89cc98330ff405ea44635ceec24713003c5d7 not found: ID does not exist" containerID="0ec8b9365bd720534aba17ca6ae89cc98330ff405ea44635ceec24713003c5d7"
Sep 16 19:41:47.711737 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.711646    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec8b9365bd720534aba17ca6ae89cc98330ff405ea44635ceec24713003c5d7"} err="failed to get container status \"0ec8b9365bd720534aba17ca6ae89cc98330ff405ea44635ceec24713003c5d7\": rpc error: code = NotFound desc = could not find container \"0ec8b9365bd720534aba17ca6ae89cc98330ff405ea44635ceec24713003c5d7\": container with ID starting with 0ec8b9365bd720534aba17ca6ae89cc98330ff405ea44635ceec24713003c5d7 not found: ID does not exist"
Sep 16 19:41:47.711737 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.711664    2582 scope.go:117] "RemoveContainer" containerID="8b60c4e8bb082969d08d87880035942f6129cff4bc1a7a07c7b76083b54e0f0b"
Sep 16 19:41:47.712000 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:41:47.711984    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b60c4e8bb082969d08d87880035942f6129cff4bc1a7a07c7b76083b54e0f0b\": container with ID starting with 8b60c4e8bb082969d08d87880035942f6129cff4bc1a7a07c7b76083b54e0f0b not found: ID does not exist" containerID="8b60c4e8bb082969d08d87880035942f6129cff4bc1a7a07c7b76083b54e0f0b"
Sep 16 19:41:47.712046 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.712005    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b60c4e8bb082969d08d87880035942f6129cff4bc1a7a07c7b76083b54e0f0b"} err="failed to get container status \"8b60c4e8bb082969d08d87880035942f6129cff4bc1a7a07c7b76083b54e0f0b\": rpc error: code = NotFound desc = could not find container \"8b60c4e8bb082969d08d87880035942f6129cff4bc1a7a07c7b76083b54e0f0b\": container with ID starting with 8b60c4e8bb082969d08d87880035942f6129cff4bc1a7a07c7b76083b54e0f0b not found: ID does not exist"
Sep 16 19:41:47.712835 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:47.712817    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5ddxsvd"]
Sep 16 19:41:48.703531 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:41:48.703497    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" path="/var/lib/kubelet/pods/361ee333-832f-445d-937a-b3a5d18f53a2/volumes"
Sep 16 19:42:21.559849 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:21.559808    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:21.651107 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:21.651058    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:27.655685 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:27.655643    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:33.183258 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:33.183163    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:33.332970 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:33.332926    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:38.080015 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:38.079978    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:38.180280 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:38.180250    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:40.629259 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.629231    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"]
Sep 16 19:42:40.629627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.629521    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main"
Sep 16 19:42:40.629627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.629531    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main"
Sep 16 19:42:40.629627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.629542    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="modelcar-init"
Sep 16 19:42:40.629627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.629548    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="modelcar-init"
Sep 16 19:42:40.629627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.629559    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="modelcar"
Sep 16 19:42:40.629627 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.629564    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="modelcar"
Sep 16 19:42:40.629823 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.629646    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="modelcar"
Sep 16 19:42:40.629823 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.629654    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="361ee333-832f-445d-937a-b3a5d18f53a2" containerName="main"
Sep 16 19:42:40.632700 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.632684    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.637021 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.636992    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 16 19:42:40.637178 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.637027    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-sjwhv\""
Sep 16 19:42:40.637178 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.637076    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 19:42:40.637178 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.637089    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:42:40.637178 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.637133    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 19:42:40.637356 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.637324    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 19:42:40.644144 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.644122    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"]
Sep 16 19:42:40.644380 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.644365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-model-cache\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.644443 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.644394    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-dshm\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.644443 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.644411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnc7k\" (UniqueName: \"kubernetes.io/projected/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-kube-api-access-fnc7k\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.644443 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.644429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-cabundle-cert\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.644635 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.644493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.644635 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.644533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-home\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.644635 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.644549    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-tmp-dir\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.644635 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.644595    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-tls-certs\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.745232 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.745211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-tls-certs\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.745383 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.745253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-model-cache\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.745383 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.745282    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-dshm\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.745383 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.745307    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnc7k\" (UniqueName: \"kubernetes.io/projected/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-kube-api-access-fnc7k\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.745383 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.745334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-cabundle-cert\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.745383 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.745376    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.745688 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.745404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-home\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.745688 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.745425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-tmp-dir\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.745807 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.745695    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-model-cache\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.745860 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.745797    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-home\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.745860 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.745812    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-tmp-dir\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.746055 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.746030    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.746159 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.746091    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-cabundle-cert\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.747701 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.747681    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-dshm\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.747875 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.747859    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-tls-certs\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.756526 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.756497    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnc7k\" (UniqueName: \"kubernetes.io/projected/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-kube-api-access-fnc7k\") pod \"custom-route-timeout-test-kserve-9479c6dcb-57htr\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:40.801425 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.801404    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:40.943697 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:40.943638    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:41.071038 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:41.071002    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:41.221586 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:41.221549    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"]
Sep 16 19:42:41.223644 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:42:41.223618    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a94a7a0_52f6_404e_a0f4_1694a973c5f4.slice/crio-f5a4fd5a7684326ef5fee48f56836b531c0a4bff340d65f993a4d4d1893989e4 WatchSource:0}: Error finding container f5a4fd5a7684326ef5fee48f56836b531c0a4bff340d65f993a4d4d1893989e4: Status 404 returned error can't find the container with id f5a4fd5a7684326ef5fee48f56836b531c0a4bff340d65f993a4d4d1893989e4
Sep 16 19:42:41.923944 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:41.923913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" event={"ID":"6a94a7a0-52f6-404e-a0f4-1694a973c5f4","Type":"ContainerStarted","Data":"ac79ddde19e9b5e8b82688dad5fa28f4d8a8a8d2ed290de8746ac813174f80f9"}
Sep 16 19:42:41.924301 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:41.923961    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" event={"ID":"6a94a7a0-52f6-404e-a0f4-1694a973c5f4","Type":"ContainerStarted","Data":"f5a4fd5a7684326ef5fee48f56836b531c0a4bff340d65f993a4d4d1893989e4"}
Sep 16 19:42:43.936254 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:43.936213    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerID="ac79ddde19e9b5e8b82688dad5fa28f4d8a8a8d2ed290de8746ac813174f80f9" exitCode=0
Sep 16 19:42:43.936694 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:43.936272    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" event={"ID":"6a94a7a0-52f6-404e-a0f4-1694a973c5f4","Type":"ContainerDied","Data":"ac79ddde19e9b5e8b82688dad5fa28f4d8a8a8d2ed290de8746ac813174f80f9"}
Sep 16 19:42:44.942875 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:44.942836    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" event={"ID":"6a94a7a0-52f6-404e-a0f4-1694a973c5f4","Type":"ContainerStarted","Data":"e4eceeba8053525d0a98894d93f1543dba92150c8bb203383e136ffe7b3ae265"}
Sep 16 19:42:44.944258 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:44.944229    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:44.969025 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:44.968951    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podStartSLOduration=2.694037446 podStartE2EDuration="4.968929785s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.274892339s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:44.965882277 +0000 UTC m=+3422.796490816" watchObservedRunningTime="2026-09-16 19:42:44.968929785 +0000 UTC m=+3422.799538329"
Sep 16 19:42:50.124569 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:50.124530    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:50.944172 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:50.944121    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:42:50.945856 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:42:50.945808    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:43:00.944194 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:00.944153    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:43:05.781930 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:05.781887    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"]
Sep 16 19:43:05.813030 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:05.812960    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:06.477213 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:06.477129    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff" podUID="d5d295a0-49a1-4118-a10d-d20e9824d24e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:43:07.334404 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:07.334363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:10.944452 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:10.944395    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:43:16.476152 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:16.476111    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff" podUID="d5d295a0-49a1-4118-a10d-d20e9824d24e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:43:20.815870 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:20.815813    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff" podUID="d5d295a0-49a1-4118-a10d-d20e9824d24e" containerName="main" containerID="cri-o://98930fc1f8418a9b3766c2967e028394aed7ca32d3c84672afc173ef59b6fd2d" gracePeriod=45
Sep 16 19:43:20.944183 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:20.944124    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:43:26.476479 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:26.476435    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff" podUID="d5d295a0-49a1-4118-a10d-d20e9824d24e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:43:26.478963 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:26.476557    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"
Sep 16 19:43:30.944810 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:30.944760    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:43:36.476931 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:36.476876    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff" podUID="d5d295a0-49a1-4118-a10d-d20e9824d24e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:43:40.944423 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:40.944363    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:43:46.476390 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:46.476333    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff" podUID="d5d295a0-49a1-4118-a10d-d20e9824d24e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:43:50.945195 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:50.945133    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:43:51.087791 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.087694    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"
Sep 16 19:43:51.138041 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.137988    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d295a0-49a1-4118-a10d-d20e9824d24e-tls-certs\") pod \"d5d295a0-49a1-4118-a10d-d20e9824d24e\" (UID: \"d5d295a0-49a1-4118-a10d-d20e9824d24e\") "
Sep 16 19:43:51.138272 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.138116    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rmm2\" (UniqueName: \"kubernetes.io/projected/d5d295a0-49a1-4118-a10d-d20e9824d24e-kube-api-access-9rmm2\") pod \"d5d295a0-49a1-4118-a10d-d20e9824d24e\" (UID: \"d5d295a0-49a1-4118-a10d-d20e9824d24e\") "
Sep 16 19:43:51.140911 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.140863    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5d295a0-49a1-4118-a10d-d20e9824d24e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5d295a0-49a1-4118-a10d-d20e9824d24e" (UID: "d5d295a0-49a1-4118-a10d-d20e9824d24e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:51.141075 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.140954    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d295a0-49a1-4118-a10d-d20e9824d24e-kube-api-access-9rmm2" (OuterVolumeSpecName: "kube-api-access-9rmm2") pod "d5d295a0-49a1-4118-a10d-d20e9824d24e" (UID: "d5d295a0-49a1-4118-a10d-d20e9824d24e"). InnerVolumeSpecName "kube-api-access-9rmm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:51.239497 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.239447    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rmm2\" (UniqueName: \"kubernetes.io/projected/d5d295a0-49a1-4118-a10d-d20e9824d24e-kube-api-access-9rmm2\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:43:51.239497 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.239487    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d295a0-49a1-4118-a10d-d20e9824d24e-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:43:51.308195 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.308152    2582 generic.go:356] "Generic (PLEG): container finished" podID="d5d295a0-49a1-4118-a10d-d20e9824d24e" containerID="98930fc1f8418a9b3766c2967e028394aed7ca32d3c84672afc173ef59b6fd2d" exitCode=0
Sep 16 19:43:51.308405 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.308211    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"
Sep 16 19:43:51.308405 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.308254    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff" event={"ID":"d5d295a0-49a1-4118-a10d-d20e9824d24e","Type":"ContainerDied","Data":"98930fc1f8418a9b3766c2967e028394aed7ca32d3c84672afc173ef59b6fd2d"}
Sep 16 19:43:51.308405 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.308334    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff" event={"ID":"d5d295a0-49a1-4118-a10d-d20e9824d24e","Type":"ContainerDied","Data":"58a172f23f32e4c5ac8dacaf6c1d343e36e92597efbbe1a8f5c5b3f7fa559901"}
Sep 16 19:43:51.308405 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.308371    2582 scope.go:117] "RemoveContainer" containerID="98930fc1f8418a9b3766c2967e028394aed7ca32d3c84672afc173ef59b6fd2d"
Sep 16 19:43:51.318987 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.318968    2582 scope.go:117] "RemoveContainer" containerID="98930fc1f8418a9b3766c2967e028394aed7ca32d3c84672afc173ef59b6fd2d"
Sep 16 19:43:51.319301 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:43:51.319280    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98930fc1f8418a9b3766c2967e028394aed7ca32d3c84672afc173ef59b6fd2d\": container with ID starting with 98930fc1f8418a9b3766c2967e028394aed7ca32d3c84672afc173ef59b6fd2d not found: ID does not exist" containerID="98930fc1f8418a9b3766c2967e028394aed7ca32d3c84672afc173ef59b6fd2d"
Sep 16 19:43:51.319380 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.319308    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98930fc1f8418a9b3766c2967e028394aed7ca32d3c84672afc173ef59b6fd2d"} err="failed to get container status \"98930fc1f8418a9b3766c2967e028394aed7ca32d3c84672afc173ef59b6fd2d\": rpc error: code = NotFound desc = could not find container \"98930fc1f8418a9b3766c2967e028394aed7ca32d3c84672afc173ef59b6fd2d\": container with ID starting with 98930fc1f8418a9b3766c2967e028394aed7ca32d3c84672afc173ef59b6fd2d not found: ID does not exist"
Sep 16 19:43:51.332629 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.332600    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"]
Sep 16 19:43:51.338863 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:51.338756    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nrxff"]
Sep 16 19:43:52.706288 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:52.706237    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5d295a0-49a1-4118-a10d-d20e9824d24e" path="/var/lib/kubelet/pods/d5d295a0-49a1-4118-a10d-d20e9824d24e/volumes"
Sep 16 19:43:55.515792 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:55.515742    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:55.620252 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:55.620205    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:55.724403 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:43:55.724360    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:00.945161 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:00.945094    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:44:01.687484 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:01.687445    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:06.929519 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:06.929423    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:07.765250 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:07.765191    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:07.825605 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:07.825553    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:10.945154 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:10.945102    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:44:18.907161 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:18.907118    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:18.979108 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:18.979061    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:20.945172 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:20.945112    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:44:30.944268 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:30.944197    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:44:40.945014 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:40.944954    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:44:50.955341 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:50.955298    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:44:50.964001 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:50.963971    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:44:56.748630 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:56.748566    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"]
Sep 16 19:44:56.753517 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:56.753424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:56.846363 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:44:56.846315    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:02.458914 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:02.458874    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:11.777910 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:11.777841    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main" containerID="cri-o://e4eceeba8053525d0a98894d93f1543dba92150c8bb203383e136ffe7b3ae265" gracePeriod=45
Sep 16 19:45:11.956009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:11.955955    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 16 19:45:12.538827 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.538795    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:45:12.659422 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.659293    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-kserve-provision-location\") pod \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") "
Sep 16 19:45:12.659422 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.659343    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-cabundle-cert\") pod \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") "
Sep 16 19:45:12.659422 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.659389    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-model-cache\") pod \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") "
Sep 16 19:45:12.659422 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.659428    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-home\") pod \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") "
Sep 16 19:45:12.659866 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.659612    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnc7k\" (UniqueName: \"kubernetes.io/projected/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-kube-api-access-fnc7k\") pod \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") "
Sep 16 19:45:12.659866 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.659686    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-dshm\") pod \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") "
Sep 16 19:45:12.659866 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.659721    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-tmp-dir\") pod \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") "
Sep 16 19:45:12.659866 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.659746    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-tls-certs\") pod \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\" (UID: \"6a94a7a0-52f6-404e-a0f4-1694a973c5f4\") "
Sep 16 19:45:12.659866 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.659765    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a94a7a0-52f6-404e-a0f4-1694a973c5f4" (UID: "6a94a7a0-52f6-404e-a0f4-1694a973c5f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:12.659866 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.659846    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a94a7a0-52f6-404e-a0f4-1694a973c5f4" (UID: "6a94a7a0-52f6-404e-a0f4-1694a973c5f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:45:12.660268 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.660219    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:45:12.660268 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.660244    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-cabundle-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:45:12.660526 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.660344    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-home" (OuterVolumeSpecName: "home") pod "6a94a7a0-52f6-404e-a0f4-1694a973c5f4" (UID: "6a94a7a0-52f6-404e-a0f4-1694a973c5f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:12.662404 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.662372    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-dshm" (OuterVolumeSpecName: "dshm") pod "6a94a7a0-52f6-404e-a0f4-1694a973c5f4" (UID: "6a94a7a0-52f6-404e-a0f4-1694a973c5f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:12.662565 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.662545    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a94a7a0-52f6-404e-a0f4-1694a973c5f4" (UID: "6a94a7a0-52f6-404e-a0f4-1694a973c5f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:45:12.662684 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.662666    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-kube-api-access-fnc7k" (OuterVolumeSpecName: "kube-api-access-fnc7k") pod "6a94a7a0-52f6-404e-a0f4-1694a973c5f4" (UID: "6a94a7a0-52f6-404e-a0f4-1694a973c5f4"). InnerVolumeSpecName "kube-api-access-fnc7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:45:12.677921 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.677875    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a94a7a0-52f6-404e-a0f4-1694a973c5f4" (UID: "6a94a7a0-52f6-404e-a0f4-1694a973c5f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:12.684662 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.684636    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a94a7a0-52f6-404e-a0f4-1694a973c5f4" (UID: "6a94a7a0-52f6-404e-a0f4-1694a973c5f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:12.734695 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.734657    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerID="e4eceeba8053525d0a98894d93f1543dba92150c8bb203383e136ffe7b3ae265" exitCode=0
Sep 16 19:45:12.734902 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.734728    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"
Sep 16 19:45:12.734902 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.734738    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" event={"ID":"6a94a7a0-52f6-404e-a0f4-1694a973c5f4","Type":"ContainerDied","Data":"e4eceeba8053525d0a98894d93f1543dba92150c8bb203383e136ffe7b3ae265"}
Sep 16 19:45:12.734902 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.734831    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr" event={"ID":"6a94a7a0-52f6-404e-a0f4-1694a973c5f4","Type":"ContainerDied","Data":"f5a4fd5a7684326ef5fee48f56836b531c0a4bff340d65f993a4d4d1893989e4"}
Sep 16 19:45:12.734902 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.734855    2582 scope.go:117] "RemoveContainer" containerID="e4eceeba8053525d0a98894d93f1543dba92150c8bb203383e136ffe7b3ae265"
Sep 16 19:45:12.744287 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.744263    2582 scope.go:117] "RemoveContainer" containerID="ac79ddde19e9b5e8b82688dad5fa28f4d8a8a8d2ed290de8746ac813174f80f9"
Sep 16 19:45:12.753098 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.753064    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"]
Sep 16 19:45:12.753881 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.753858    2582 scope.go:117] "RemoveContainer" containerID="e4eceeba8053525d0a98894d93f1543dba92150c8bb203383e136ffe7b3ae265"
Sep 16 19:45:12.754765 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:45:12.754330    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4eceeba8053525d0a98894d93f1543dba92150c8bb203383e136ffe7b3ae265\": container with ID starting with e4eceeba8053525d0a98894d93f1543dba92150c8bb203383e136ffe7b3ae265 not found: ID does not exist" containerID="e4eceeba8053525d0a98894d93f1543dba92150c8bb203383e136ffe7b3ae265"
Sep 16 19:45:12.754765 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.754399    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4eceeba8053525d0a98894d93f1543dba92150c8bb203383e136ffe7b3ae265"} err="failed to get container status \"e4eceeba8053525d0a98894d93f1543dba92150c8bb203383e136ffe7b3ae265\": rpc error: code = NotFound desc = could not find container \"e4eceeba8053525d0a98894d93f1543dba92150c8bb203383e136ffe7b3ae265\": container with ID starting with e4eceeba8053525d0a98894d93f1543dba92150c8bb203383e136ffe7b3ae265 not found: ID does not exist"
Sep 16 19:45:12.754765 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.754428    2582 scope.go:117] "RemoveContainer" containerID="ac79ddde19e9b5e8b82688dad5fa28f4d8a8a8d2ed290de8746ac813174f80f9"
Sep 16 19:45:12.755014 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:45:12.754894    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac79ddde19e9b5e8b82688dad5fa28f4d8a8a8d2ed290de8746ac813174f80f9\": container with ID starting with ac79ddde19e9b5e8b82688dad5fa28f4d8a8a8d2ed290de8746ac813174f80f9 not found: ID does not exist" containerID="ac79ddde19e9b5e8b82688dad5fa28f4d8a8a8d2ed290de8746ac813174f80f9"
Sep 16 19:45:12.755014 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.754928    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac79ddde19e9b5e8b82688dad5fa28f4d8a8a8d2ed290de8746ac813174f80f9"} err="failed to get container status \"ac79ddde19e9b5e8b82688dad5fa28f4d8a8a8d2ed290de8746ac813174f80f9\": rpc error: code = NotFound desc = could not find container \"ac79ddde19e9b5e8b82688dad5fa28f4d8a8a8d2ed290de8746ac813174f80f9\": container with ID starting with ac79ddde19e9b5e8b82688dad5fa28f4d8a8a8d2ed290de8746ac813174f80f9 not found: ID does not exist"
Sep 16 19:45:12.756750 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.756730    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9479c6dcb-57htr"]
Sep 16 19:45:12.761391 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.761369    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:45:12.761457 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.761394    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:45:12.761457 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.761407    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnc7k\" (UniqueName: \"kubernetes.io/projected/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-kube-api-access-fnc7k\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:45:12.761457 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.761416    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:45:12.761457 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.761447    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:45:12.761457 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:12.761457    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a94a7a0-52f6-404e-a0f4-1694a973c5f4-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:45:14.704220 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:14.704177    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" path="/var/lib/kubelet/pods/6a94a7a0-52f6-404e-a0f4-1694a973c5f4/volumes"
Sep 16 19:45:46.661770 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.661738    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:46.767152 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.767119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:46.971351 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.971321    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"]
Sep 16 19:45:46.971726 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.971710    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main"
Sep 16 19:45:46.971779 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.971729    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main"
Sep 16 19:45:46.971779 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.971754    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d295a0-49a1-4118-a10d-d20e9824d24e" containerName="main"
Sep 16 19:45:46.971779 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.971762    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d295a0-49a1-4118-a10d-d20e9824d24e" containerName="main"
Sep 16 19:45:46.971779 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.971769    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="storage-initializer"
Sep 16 19:45:46.971779 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.971774    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="storage-initializer"
Sep 16 19:45:46.971936 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.971824    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d295a0-49a1-4118-a10d-d20e9824d24e" containerName="main"
Sep 16 19:45:46.971936 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.971835    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a94a7a0-52f6-404e-a0f4-1694a973c5f4" containerName="main"
Sep 16 19:45:46.975786 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.975770    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:46.978941 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.978918    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-wcxjc\""
Sep 16 19:45:46.979088 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.978926    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:45:46.980220 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.980205    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 19:45:46.980339 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.980321    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 19:45:46.996325 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:46.996294    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"]
Sep 16 19:45:47.033166 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.033124    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:47.064644 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.064606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.064644 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.064647    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bb5314-2297-48db-80ae-8c164adebf4c-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.064855 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.064677    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bb5314-2297-48db-80ae-8c164adebf4c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.064855 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.064728    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.064855 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.064760    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w59jm\" (UniqueName: \"kubernetes.io/projected/90bb5314-2297-48db-80ae-8c164adebf4c-kube-api-access-w59jm\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.064855 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.064783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.064855 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.064802    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.064855 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.064846    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.065053 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.064883    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bb5314-2297-48db-80ae-8c164adebf4c-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.125487 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.125449    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:47.165678 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.165642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.165819 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.165687    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bb5314-2297-48db-80ae-8c164adebf4c-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.165819 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.165745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.165819 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.165768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bb5314-2297-48db-80ae-8c164adebf4c-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.165819 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.165795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bb5314-2297-48db-80ae-8c164adebf4c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.166030 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.165833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.166030 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.165858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w59jm\" (UniqueName: \"kubernetes.io/projected/90bb5314-2297-48db-80ae-8c164adebf4c-kube-api-access-w59jm\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.166030 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.165889    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.166030 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.165917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.166236 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.166137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.166236 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.166157    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.166236 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.166202    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.166492 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.166472    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.166754 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.166731    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bb5314-2297-48db-80ae-8c164adebf4c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.168408 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.168382    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.168540 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.168526    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bb5314-2297-48db-80ae-8c164adebf4c-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.176673 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.176649    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w59jm\" (UniqueName: \"kubernetes.io/projected/90bb5314-2297-48db-80ae-8c164adebf4c-kube-api-access-w59jm\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.176770 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.176729    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bb5314-2297-48db-80ae-8c164adebf4c-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-rfh9k\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.286642 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.286605    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:47.572321 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.572291    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"]
Sep 16 19:45:47.575194 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:45:47.575159    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bb5314_2297_48db_80ae_8c164adebf4c.slice/crio-4bb441de1e43718655bf91bf3dbfe524cbbe457b91953175e9af518ab51659d3 WatchSource:0}: Error finding container 4bb441de1e43718655bf91bf3dbfe524cbbe457b91953175e9af518ab51659d3: Status 404 returned error can't find the container with id 4bb441de1e43718655bf91bf3dbfe524cbbe457b91953175e9af518ab51659d3
Sep 16 19:45:47.577835 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.577802    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:45:47.577898 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.577873    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:45:47.577936 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.577904    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:45:47.896476 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.896367    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k" event={"ID":"90bb5314-2297-48db-80ae-8c164adebf4c","Type":"ContainerStarted","Data":"4b719b41f17e0e90b1d7d66c54b92b4471a1e207117c2c58ea7ac705d8a38fec"}
Sep 16 19:45:47.896476 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.896431    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k" event={"ID":"90bb5314-2297-48db-80ae-8c164adebf4c","Type":"ContainerStarted","Data":"4bb441de1e43718655bf91bf3dbfe524cbbe457b91953175e9af518ab51659d3"}
Sep 16 19:45:47.919114 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:47.919052    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k" podStartSLOduration=1.919035208 podStartE2EDuration="1.919035208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:45:47.916443728 +0000 UTC m=+3605.747052281" watchObservedRunningTime="2026-09-16 19:45:47.919035208 +0000 UTC m=+3605.749643753"
Sep 16 19:45:48.287261 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:48.287220    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:48.292469 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:48.292444    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:48.901498 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:48.901458    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:48.902690 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:48.902663    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:45:50.467152 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:50.467122    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:53.937742 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:53.937686    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:55.653812 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.653725    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"]
Sep 16 19:45:55.657340 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.657318    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.661512 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.661486    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 19:45:55.661709 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.661641    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 16 19:45:55.661709 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.661674    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-qmm49\""
Sep 16 19:45:55.661983 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.661969    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:45:55.689714 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.689682    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"]
Sep 16 19:45:55.700500 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.700461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:55.764714 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.764676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-home\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.764714 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.764714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-tmp-dir\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.765009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.764750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvljm\" (UniqueName: \"kubernetes.io/projected/a54270fe-c658-436f-b4d7-784175aa319b-kube-api-access-hvljm\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.765009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.764833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-dshm\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.765009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.764885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54270fe-c658-436f-b4d7-784175aa319b-cabundle-cert\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.765009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.764927    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-model-cache\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.765009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.764947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-kserve-provision-location\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.765009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.764972    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54270fe-c658-436f-b4d7-784175aa319b-tls-certs\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.865628 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.865555    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvljm\" (UniqueName: \"kubernetes.io/projected/a54270fe-c658-436f-b4d7-784175aa319b-kube-api-access-hvljm\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.865859 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.865656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-dshm\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.865859 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.865729    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54270fe-c658-436f-b4d7-784175aa319b-cabundle-cert\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.865859 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.865790    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-model-cache\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.865859 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.865815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-kserve-provision-location\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.865859 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.865858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54270fe-c658-436f-b4d7-784175aa319b-tls-certs\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.866151 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.865942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-home\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.866151 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.865970    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-tmp-dir\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.866261 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.866188    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-model-cache\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.866398 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.866369    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-kserve-provision-location\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.866466 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.866398    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-tmp-dir\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.866589 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.866555    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-home\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.866933 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.866913    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54270fe-c658-436f-b4d7-784175aa319b-cabundle-cert\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.868961 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.868930    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-dshm\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.869347 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.869326    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54270fe-c658-436f-b4d7-784175aa319b-tls-certs\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.877137 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.877091    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvljm\" (UniqueName: \"kubernetes.io/projected/a54270fe-c658-436f-b4d7-784175aa319b-kube-api-access-hvljm\") pod \"router-with-refs-test-kserve-5dcdb8d756-gbf5m\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:55.969806 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:55.969704    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:45:56.143837 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:56.143802    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:56.220480 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:56.220362    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:56.261084 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:56.261035    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"]
Sep 16 19:45:56.266488 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:45:56.266455    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54270fe_c658_436f_b4d7_784175aa319b.slice/crio-b7e02084421b2969d6070252cbc06489077cf69b7cb01714269949dfed563068 WatchSource:0}: Error finding container b7e02084421b2969d6070252cbc06489077cf69b7cb01714269949dfed563068: Status 404 returned error can't find the container with id b7e02084421b2969d6070252cbc06489077cf69b7cb01714269949dfed563068
Sep 16 19:45:56.950594 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:56.950519    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" event={"ID":"a54270fe-c658-436f-b4d7-784175aa319b","Type":"ContainerStarted","Data":"0cb93e19174e6f4d0b2871d578413cef6ed393d4ca2b5f3cacc17fd7e90fb772"}
Sep 16 19:45:56.951044 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:56.950623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" event={"ID":"a54270fe-c658-436f-b4d7-784175aa319b","Type":"ContainerStarted","Data":"b7e02084421b2969d6070252cbc06489077cf69b7cb01714269949dfed563068"}
Sep 16 19:45:58.961161 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:58.961117    2582 generic.go:356] "Generic (PLEG): container finished" podID="a54270fe-c658-436f-b4d7-784175aa319b" containerID="0cb93e19174e6f4d0b2871d578413cef6ed393d4ca2b5f3cacc17fd7e90fb772" exitCode=0
Sep 16 19:45:58.961736 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:58.961201    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" event={"ID":"a54270fe-c658-436f-b4d7-784175aa319b","Type":"ContainerDied","Data":"0cb93e19174e6f4d0b2871d578413cef6ed393d4ca2b5f3cacc17fd7e90fb772"}
Sep 16 19:45:58.963215 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:58.963187    2582 generic.go:356] "Generic (PLEG): container finished" podID="90bb5314-2297-48db-80ae-8c164adebf4c" containerID="4b719b41f17e0e90b1d7d66c54b92b4471a1e207117c2c58ea7ac705d8a38fec" exitCode=0
Sep 16 19:45:58.963342 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:58.963270    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k" event={"ID":"90bb5314-2297-48db-80ae-8c164adebf4c","Type":"ContainerDied","Data":"4b719b41f17e0e90b1d7d66c54b92b4471a1e207117c2c58ea7ac705d8a38fec"}
Sep 16 19:45:58.963697 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:58.963682    2582 scope.go:117] "RemoveContainer" containerID="4b719b41f17e0e90b1d7d66c54b92b4471a1e207117c2c58ea7ac705d8a38fec"
Sep 16 19:45:58.964669 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:58.964636    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:45:58.964754 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:58.964713    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:45:58.964793 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:58.964756    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:45:59.972018 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:59.971940    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" event={"ID":"a54270fe-c658-436f-b4d7-784175aa319b","Type":"ContainerStarted","Data":"3415e7021d6c1f37501ebe069eb78e36d5dae11fa843ae54ebbc4561094f13b9"}
Sep 16 19:45:59.974374 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:45:59.974347    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k" event={"ID":"90bb5314-2297-48db-80ae-8c164adebf4c","Type":"ContainerStarted","Data":"4578748fc56f74761aa4beba6dd1be6d3664980cbb4c6cfaf86b0aa462fa7dd5"}
Sep 16 19:46:00.010039 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:00.009961    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" podStartSLOduration=2.5231862659999997 podStartE2EDuration="5.009937534s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.486751268s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:46:00.005525736 +0000 UTC m=+3617.836134282" watchObservedRunningTime="2026-09-16 19:46:00.009937534 +0000 UTC m=+3617.840546084"
Sep 16 19:46:00.287408 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:00.287369    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:46:00.288616 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:00.288588    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k" podUID="90bb5314-2297-48db-80ae-8c164adebf4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 16 19:46:00.320621 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:00.320533    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:00.381324 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:00.381284    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:00.970505 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:00.970464    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:46:01.287502 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:01.287464    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k" podUID="90bb5314-2297-48db-80ae-8c164adebf4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 16 19:46:02.287955 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:02.287909    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k" podUID="90bb5314-2297-48db-80ae-8c164adebf4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 16 19:46:03.287511 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:03.287466    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k" podUID="90bb5314-2297-48db-80ae-8c164adebf4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 16 19:46:03.902445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:03.902389    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:46:04.288027 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:04.287977    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k" podUID="90bb5314-2297-48db-80ae-8c164adebf4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 16 19:46:05.287746 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:05.287690    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k" podUID="90bb5314-2297-48db-80ae-8c164adebf4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 16 19:46:05.970555 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:05.970501    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:46:05.972484 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:05.972445    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:46:06.291311 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:06.291275    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:46:06.292336 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:06.292314    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:46:12.562111 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:12.562068    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:15.970266 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:15.970224    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:46:20.233486 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:20.233438    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:25.971217 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:25.971159    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:46:28.867495 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:28.867454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:28.985112 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:28.985075    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:29.088067 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:29.088035    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:34.805944 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:34.805901    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:34.905812 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:34.905776    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:35.970319 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:35.970274    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:46:41.755217 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:41.755173    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:45.970531 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:45.970470    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:46:55.970604 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:46:55.970515    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:47:05.970157 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:47:05.970069    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:47:15.970401 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:47:15.970350    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:47:25.971098 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:47:25.971035    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:47:35.970942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:47:35.970891    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:47:45.970868 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:47:45.970808    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:47:47.044036 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:47:47.043998    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:47.108752 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:47:47.108720    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:55.970414 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:47:55.970366    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 16 19:48:05.982242 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:05.982194    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:48:05.991899 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:05.991856    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:48:19.989200 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:19.989155    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:21.291988 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:21.291938    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:30.045330 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:30.045287    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:50.161352 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:50.161209    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:50.263389 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:50.263345    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:50.362825 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:50.362785    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:50.435673 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:50.435540    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:50.497523 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:50.497478    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:52.973485 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:52.973438    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:53.035941 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:53.035904    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:53.574397 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:53.574343    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:56.333139 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:56.333093    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:56.404043 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:48:56.403984    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:17.960693 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:49:17.960657    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:15.182424 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:15.182392    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:15.190332 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:15.190304    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"]
Sep 16 19:50:15.275116 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:15.275086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:30.213399 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.213318    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main" containerID="cri-o://3415e7021d6c1f37501ebe069eb78e36d5dae11fa843ae54ebbc4561094f13b9" gracePeriod=45
Sep 16 19:50:30.889191 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.889159    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:50:30.961326 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.961211    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvljm\" (UniqueName: \"kubernetes.io/projected/a54270fe-c658-436f-b4d7-784175aa319b-kube-api-access-hvljm\") pod \"a54270fe-c658-436f-b4d7-784175aa319b\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") "
Sep 16 19:50:30.961326 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.961282    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54270fe-c658-436f-b4d7-784175aa319b-tls-certs\") pod \"a54270fe-c658-436f-b4d7-784175aa319b\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") "
Sep 16 19:50:30.961326 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.961320    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-model-cache\") pod \"a54270fe-c658-436f-b4d7-784175aa319b\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") "
Sep 16 19:50:30.961688 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.961344    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-dshm\") pod \"a54270fe-c658-436f-b4d7-784175aa319b\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") "
Sep 16 19:50:30.961688 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.961376    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54270fe-c658-436f-b4d7-784175aa319b-cabundle-cert\") pod \"a54270fe-c658-436f-b4d7-784175aa319b\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") "
Sep 16 19:50:30.961688 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.961429    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-tmp-dir\") pod \"a54270fe-c658-436f-b4d7-784175aa319b\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") "
Sep 16 19:50:30.961688 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.961474    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-home\") pod \"a54270fe-c658-436f-b4d7-784175aa319b\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") "
Sep 16 19:50:30.961688 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.961501    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-kserve-provision-location\") pod \"a54270fe-c658-436f-b4d7-784175aa319b\" (UID: \"a54270fe-c658-436f-b4d7-784175aa319b\") "
Sep 16 19:50:30.961957 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.961791    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54270fe-c658-436f-b4d7-784175aa319b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a54270fe-c658-436f-b4d7-784175aa319b" (UID: "a54270fe-c658-436f-b4d7-784175aa319b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:50:30.961957 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.961889    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a54270fe-c658-436f-b4d7-784175aa319b" (UID: "a54270fe-c658-436f-b4d7-784175aa319b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:30.962269 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.962205    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-home" (OuterVolumeSpecName: "home") pod "a54270fe-c658-436f-b4d7-784175aa319b" (UID: "a54270fe-c658-436f-b4d7-784175aa319b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:30.964023 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.963990    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-dshm" (OuterVolumeSpecName: "dshm") pod "a54270fe-c658-436f-b4d7-784175aa319b" (UID: "a54270fe-c658-436f-b4d7-784175aa319b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:30.964162 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.964112    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54270fe-c658-436f-b4d7-784175aa319b-kube-api-access-hvljm" (OuterVolumeSpecName: "kube-api-access-hvljm") pod "a54270fe-c658-436f-b4d7-784175aa319b" (UID: "a54270fe-c658-436f-b4d7-784175aa319b"). InnerVolumeSpecName "kube-api-access-hvljm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:50:30.964261 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.964239    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54270fe-c658-436f-b4d7-784175aa319b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a54270fe-c658-436f-b4d7-784175aa319b" (UID: "a54270fe-c658-436f-b4d7-784175aa319b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:50:30.975756 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.975719    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a54270fe-c658-436f-b4d7-784175aa319b" (UID: "a54270fe-c658-436f-b4d7-784175aa319b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:30.987138 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:30.987099    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a54270fe-c658-436f-b4d7-784175aa319b" (UID: "a54270fe-c658-436f-b4d7-784175aa319b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:31.062408 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.062359    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:50:31.062408 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.062408    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54270fe-c658-436f-b4d7-784175aa319b-cabundle-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:50:31.062408 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.062422    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:50:31.062694 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.062432    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:50:31.062694 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.062445    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:50:31.062694 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.062456    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvljm\" (UniqueName: \"kubernetes.io/projected/a54270fe-c658-436f-b4d7-784175aa319b-kube-api-access-hvljm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:50:31.062694 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.062467    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54270fe-c658-436f-b4d7-784175aa319b-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:50:31.062694 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.062477    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54270fe-c658-436f-b4d7-784175aa319b-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:50:31.313196 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.313157    2582 generic.go:356] "Generic (PLEG): container finished" podID="a54270fe-c658-436f-b4d7-784175aa319b" containerID="3415e7021d6c1f37501ebe069eb78e36d5dae11fa843ae54ebbc4561094f13b9" exitCode=0
Sep 16 19:50:31.313652 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.313214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" event={"ID":"a54270fe-c658-436f-b4d7-784175aa319b","Type":"ContainerDied","Data":"3415e7021d6c1f37501ebe069eb78e36d5dae11fa843ae54ebbc4561094f13b9"}
Sep 16 19:50:31.313652 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.313260    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m" event={"ID":"a54270fe-c658-436f-b4d7-784175aa319b","Type":"ContainerDied","Data":"b7e02084421b2969d6070252cbc06489077cf69b7cb01714269949dfed563068"}
Sep 16 19:50:31.313652 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.313283    2582 scope.go:117] "RemoveContainer" containerID="3415e7021d6c1f37501ebe069eb78e36d5dae11fa843ae54ebbc4561094f13b9"
Sep 16 19:50:31.313652 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.313293    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"
Sep 16 19:50:31.325812 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.325783    2582 scope.go:117] "RemoveContainer" containerID="0cb93e19174e6f4d0b2871d578413cef6ed393d4ca2b5f3cacc17fd7e90fb772"
Sep 16 19:50:31.334016 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.333998    2582 scope.go:117] "RemoveContainer" containerID="3415e7021d6c1f37501ebe069eb78e36d5dae11fa843ae54ebbc4561094f13b9"
Sep 16 19:50:31.334303 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:50:31.334282    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3415e7021d6c1f37501ebe069eb78e36d5dae11fa843ae54ebbc4561094f13b9\": container with ID starting with 3415e7021d6c1f37501ebe069eb78e36d5dae11fa843ae54ebbc4561094f13b9 not found: ID does not exist" containerID="3415e7021d6c1f37501ebe069eb78e36d5dae11fa843ae54ebbc4561094f13b9"
Sep 16 19:50:31.334371 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.334313    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3415e7021d6c1f37501ebe069eb78e36d5dae11fa843ae54ebbc4561094f13b9"} err="failed to get container status \"3415e7021d6c1f37501ebe069eb78e36d5dae11fa843ae54ebbc4561094f13b9\": rpc error: code = NotFound desc = could not find container \"3415e7021d6c1f37501ebe069eb78e36d5dae11fa843ae54ebbc4561094f13b9\": container with ID starting with 3415e7021d6c1f37501ebe069eb78e36d5dae11fa843ae54ebbc4561094f13b9 not found: ID does not exist"
Sep 16 19:50:31.334371 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.334331    2582 scope.go:117] "RemoveContainer" containerID="0cb93e19174e6f4d0b2871d578413cef6ed393d4ca2b5f3cacc17fd7e90fb772"
Sep 16 19:50:31.334608 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:50:31.334588    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb93e19174e6f4d0b2871d578413cef6ed393d4ca2b5f3cacc17fd7e90fb772\": container with ID starting with 0cb93e19174e6f4d0b2871d578413cef6ed393d4ca2b5f3cacc17fd7e90fb772 not found: ID does not exist" containerID="0cb93e19174e6f4d0b2871d578413cef6ed393d4ca2b5f3cacc17fd7e90fb772"
Sep 16 19:50:31.334659 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.334616    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb93e19174e6f4d0b2871d578413cef6ed393d4ca2b5f3cacc17fd7e90fb772"} err="failed to get container status \"0cb93e19174e6f4d0b2871d578413cef6ed393d4ca2b5f3cacc17fd7e90fb772\": rpc error: code = NotFound desc = could not find container \"0cb93e19174e6f4d0b2871d578413cef6ed393d4ca2b5f3cacc17fd7e90fb772\": container with ID starting with 0cb93e19174e6f4d0b2871d578413cef6ed393d4ca2b5f3cacc17fd7e90fb772 not found: ID does not exist"
Sep 16 19:50:31.337804 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.337781    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"]
Sep 16 19:50:31.341858 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:31.341839    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5dcdb8d756-gbf5m"]
Sep 16 19:50:32.705389 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:50:32.705352    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a54270fe-c658-436f-b4d7-784175aa319b" path="/var/lib/kubelet/pods/a54270fe-c658-436f-b4d7-784175aa319b/volumes"
Sep 16 19:51:05.305686 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:05.305645    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:05.398475 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:05.398436    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:10.255005 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:10.254959    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"]
Sep 16 19:51:10.255477 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:10.255226    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k" podUID="90bb5314-2297-48db-80ae-8c164adebf4c" containerName="istio-proxy" containerID="cri-o://4578748fc56f74761aa4beba6dd1be6d3664980cbb4c6cfaf86b0aa462fa7dd5" gracePeriod=30
Sep 16 19:51:10.305773 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:10.305737    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:15.507414 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.507383    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:51:15.518525 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.518494    2582 generic.go:356] "Generic (PLEG): container finished" podID="90bb5314-2297-48db-80ae-8c164adebf4c" containerID="4578748fc56f74761aa4beba6dd1be6d3664980cbb4c6cfaf86b0aa462fa7dd5" exitCode=0
Sep 16 19:51:15.518705 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.518541    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"
Sep 16 19:51:15.518705 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.518544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k" event={"ID":"90bb5314-2297-48db-80ae-8c164adebf4c","Type":"ContainerDied","Data":"4578748fc56f74761aa4beba6dd1be6d3664980cbb4c6cfaf86b0aa462fa7dd5"}
Sep 16 19:51:15.518705 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.518636    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k" event={"ID":"90bb5314-2297-48db-80ae-8c164adebf4c","Type":"ContainerDied","Data":"4bb441de1e43718655bf91bf3dbfe524cbbe457b91953175e9af518ab51659d3"}
Sep 16 19:51:15.518705 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.518664    2582 scope.go:117] "RemoveContainer" containerID="4578748fc56f74761aa4beba6dd1be6d3664980cbb4c6cfaf86b0aa462fa7dd5"
Sep 16 19:51:15.527811 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.527787    2582 scope.go:117] "RemoveContainer" containerID="4b719b41f17e0e90b1d7d66c54b92b4471a1e207117c2c58ea7ac705d8a38fec"
Sep 16 19:51:15.538703 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.538676    2582 scope.go:117] "RemoveContainer" containerID="4578748fc56f74761aa4beba6dd1be6d3664980cbb4c6cfaf86b0aa462fa7dd5"
Sep 16 19:51:15.539364 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:51:15.539331    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4578748fc56f74761aa4beba6dd1be6d3664980cbb4c6cfaf86b0aa462fa7dd5\": container with ID starting with 4578748fc56f74761aa4beba6dd1be6d3664980cbb4c6cfaf86b0aa462fa7dd5 not found: ID does not exist" containerID="4578748fc56f74761aa4beba6dd1be6d3664980cbb4c6cfaf86b0aa462fa7dd5"
Sep 16 19:51:15.539468 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.539375    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4578748fc56f74761aa4beba6dd1be6d3664980cbb4c6cfaf86b0aa462fa7dd5"} err="failed to get container status \"4578748fc56f74761aa4beba6dd1be6d3664980cbb4c6cfaf86b0aa462fa7dd5\": rpc error: code = NotFound desc = could not find container \"4578748fc56f74761aa4beba6dd1be6d3664980cbb4c6cfaf86b0aa462fa7dd5\": container with ID starting with 4578748fc56f74761aa4beba6dd1be6d3664980cbb4c6cfaf86b0aa462fa7dd5 not found: ID does not exist"
Sep 16 19:51:15.539468 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.539402    2582 scope.go:117] "RemoveContainer" containerID="4b719b41f17e0e90b1d7d66c54b92b4471a1e207117c2c58ea7ac705d8a38fec"
Sep 16 19:51:15.539859 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:51:15.539825    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b719b41f17e0e90b1d7d66c54b92b4471a1e207117c2c58ea7ac705d8a38fec\": container with ID starting with 4b719b41f17e0e90b1d7d66c54b92b4471a1e207117c2c58ea7ac705d8a38fec not found: ID does not exist" containerID="4b719b41f17e0e90b1d7d66c54b92b4471a1e207117c2c58ea7ac705d8a38fec"
Sep 16 19:51:15.539968 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.539869    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b719b41f17e0e90b1d7d66c54b92b4471a1e207117c2c58ea7ac705d8a38fec"} err="failed to get container status \"4b719b41f17e0e90b1d7d66c54b92b4471a1e207117c2c58ea7ac705d8a38fec\": rpc error: code = NotFound desc = could not find container \"4b719b41f17e0e90b1d7d66c54b92b4471a1e207117c2c58ea7ac705d8a38fec\": container with ID starting with 4b719b41f17e0e90b1d7d66c54b92b4471a1e207117c2c58ea7ac705d8a38fec not found: ID does not exist"
Sep 16 19:51:15.559265 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.559227    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w59jm\" (UniqueName: \"kubernetes.io/projected/90bb5314-2297-48db-80ae-8c164adebf4c-kube-api-access-w59jm\") pod \"90bb5314-2297-48db-80ae-8c164adebf4c\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") "
Sep 16 19:51:15.559440 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.559305    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-workload-socket\") pod \"90bb5314-2297-48db-80ae-8c164adebf4c\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") "
Sep 16 19:51:15.559440 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.559332    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bb5314-2297-48db-80ae-8c164adebf4c-istiod-ca-cert\") pod \"90bb5314-2297-48db-80ae-8c164adebf4c\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") "
Sep 16 19:51:15.559440 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.559357    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-workload-certs\") pod \"90bb5314-2297-48db-80ae-8c164adebf4c\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") "
Sep 16 19:51:15.559440 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.559375    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-istio-data\") pod \"90bb5314-2297-48db-80ae-8c164adebf4c\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") "
Sep 16 19:51:15.559440 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.559419    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bb5314-2297-48db-80ae-8c164adebf4c-istio-podinfo\") pod \"90bb5314-2297-48db-80ae-8c164adebf4c\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") "
Sep 16 19:51:15.559840 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.559447    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bb5314-2297-48db-80ae-8c164adebf4c-istio-token\") pod \"90bb5314-2297-48db-80ae-8c164adebf4c\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") "
Sep 16 19:51:15.559840 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.559465    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-credential-socket\") pod \"90bb5314-2297-48db-80ae-8c164adebf4c\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") "
Sep 16 19:51:15.559840 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.559481    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-istio-envoy\") pod \"90bb5314-2297-48db-80ae-8c164adebf4c\" (UID: \"90bb5314-2297-48db-80ae-8c164adebf4c\") "
Sep 16 19:51:15.560078 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.559976    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "90bb5314-2297-48db-80ae-8c164adebf4c" (UID: "90bb5314-2297-48db-80ae-8c164adebf4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:15.560078 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.560024    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90bb5314-2297-48db-80ae-8c164adebf4c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "90bb5314-2297-48db-80ae-8c164adebf4c" (UID: "90bb5314-2297-48db-80ae-8c164adebf4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:51:15.560489 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.560452    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "90bb5314-2297-48db-80ae-8c164adebf4c" (UID: "90bb5314-2297-48db-80ae-8c164adebf4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:15.561606 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.560662    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "90bb5314-2297-48db-80ae-8c164adebf4c" (UID: "90bb5314-2297-48db-80ae-8c164adebf4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:15.561606 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.560725    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-istio-data" (OuterVolumeSpecName: "istio-data") pod "90bb5314-2297-48db-80ae-8c164adebf4c" (UID: "90bb5314-2297-48db-80ae-8c164adebf4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:15.561954 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.561930    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bb5314-2297-48db-80ae-8c164adebf4c-kube-api-access-w59jm" (OuterVolumeSpecName: "kube-api-access-w59jm") pod "90bb5314-2297-48db-80ae-8c164adebf4c" (UID: "90bb5314-2297-48db-80ae-8c164adebf4c"). InnerVolumeSpecName "kube-api-access-w59jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:51:15.562554 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.562527    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "90bb5314-2297-48db-80ae-8c164adebf4c" (UID: "90bb5314-2297-48db-80ae-8c164adebf4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:51:15.562716 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.562693    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90bb5314-2297-48db-80ae-8c164adebf4c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "90bb5314-2297-48db-80ae-8c164adebf4c" (UID: "90bb5314-2297-48db-80ae-8c164adebf4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:51:15.562958 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.562937    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bb5314-2297-48db-80ae-8c164adebf4c-istio-token" (OuterVolumeSpecName: "istio-token") pod "90bb5314-2297-48db-80ae-8c164adebf4c" (UID: "90bb5314-2297-48db-80ae-8c164adebf4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:51:15.660170 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.660127    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w59jm\" (UniqueName: \"kubernetes.io/projected/90bb5314-2297-48db-80ae-8c164adebf4c-kube-api-access-w59jm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:51:15.660170 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.660158    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-workload-socket\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:51:15.660170 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.660169    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bb5314-2297-48db-80ae-8c164adebf4c-istiod-ca-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:51:15.660170 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.660177    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-workload-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:51:15.660445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.660186    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-istio-data\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:51:15.660445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.660195    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bb5314-2297-48db-80ae-8c164adebf4c-istio-podinfo\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:51:15.660445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.660203    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bb5314-2297-48db-80ae-8c164adebf4c-istio-token\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:51:15.660445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.660212    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-credential-socket\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:51:15.660445 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.660219    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bb5314-2297-48db-80ae-8c164adebf4c-istio-envoy\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:51:15.845697 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.845662    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"]
Sep 16 19:51:15.849588 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:15.849549    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-rfh9k"]
Sep 16 19:51:16.704168 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:16.704133    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90bb5314-2297-48db-80ae-8c164adebf4c" path="/var/lib/kubelet/pods/90bb5314-2297-48db-80ae-8c164adebf4c/volumes"
Sep 16 19:51:24.862005 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.861952    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"]
Sep 16 19:51:24.862529 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.862480    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="storage-initializer"
Sep 16 19:51:24.862529 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.862504    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="storage-initializer"
Sep 16 19:51:24.862529 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.862525    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90bb5314-2297-48db-80ae-8c164adebf4c" containerName="istio-proxy"
Sep 16 19:51:24.862750 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.862534    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90bb5314-2297-48db-80ae-8c164adebf4c" containerName="istio-proxy"
Sep 16 19:51:24.862750 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.862554    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main"
Sep 16 19:51:24.862750 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.862563    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main"
Sep 16 19:51:24.862750 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.862716    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="90bb5314-2297-48db-80ae-8c164adebf4c" containerName="istio-proxy"
Sep 16 19:51:24.862750 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.862736    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a54270fe-c658-436f-b4d7-784175aa319b" containerName="main"
Sep 16 19:51:24.863003 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.862842    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90bb5314-2297-48db-80ae-8c164adebf4c" containerName="istio-proxy"
Sep 16 19:51:24.863003 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.862853    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90bb5314-2297-48db-80ae-8c164adebf4c" containerName="istio-proxy"
Sep 16 19:51:24.863003 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.862939    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="90bb5314-2297-48db-80ae-8c164adebf4c" containerName="istio-proxy"
Sep 16 19:51:24.868077 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.868052    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:24.871327 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.871298    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 19:51:24.871484 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.871298    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 19:51:24.871484 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.871298    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-5lvhp\""
Sep 16 19:51:24.871484 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.871358    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-gphrq\""
Sep 16 19:51:24.871484 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.871357    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 19:51:24.871484 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.871299    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:51:24.872548 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.872530    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 19:51:24.879046 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.879022    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"]
Sep 16 19:51:24.934959 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.934921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4hfg\" (UniqueName: \"kubernetes.io/projected/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-kube-api-access-z4hfg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:24.935193 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.934984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:24.935193 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.935027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:24.935193 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.935055    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:24.935193 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.935122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:24.935193 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.935186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:24.935424 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.935210    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:24.935424 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.935232    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:24.936866 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:24.936837    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:25.036462 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.036429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.036599 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.036472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.036599 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.036495    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.036599 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.036526    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.036599 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.036546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.036797 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.036567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.036797 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.036698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4hfg\" (UniqueName: \"kubernetes.io/projected/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-kube-api-access-z4hfg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.036797 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.036741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.036950 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.036917    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.037009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.036983    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.037078 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.037060    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.037133 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.037105    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.037502 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.037476    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.039420 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.039396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.039681 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.039661    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.046168 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.046140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4hfg\" (UniqueName: \"kubernetes.io/projected/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-kube-api-access-z4hfg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.179391 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.179288    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:25.183800 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.183657    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"]
Sep 16 19:51:25.187496 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.187470    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"
Sep 16 19:51:25.190781 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.190746    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-kqp82\""
Sep 16 19:51:25.195419 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.195390    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"]
Sep 16 19:51:25.211215 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.211183    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:25.239750 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.239668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg\" (UID: \"493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"
Sep 16 19:51:25.239925 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.239812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr7tt\" (UniqueName: \"kubernetes.io/projected/493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b-kube-api-access-gr7tt\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg\" (UID: \"493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"
Sep 16 19:51:25.340462 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.340417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg\" (UID: \"493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"
Sep 16 19:51:25.340462 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.340466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr7tt\" (UniqueName: \"kubernetes.io/projected/493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b-kube-api-access-gr7tt\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg\" (UID: \"493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"
Sep 16 19:51:25.343309 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.343284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg\" (UID: \"493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"
Sep 16 19:51:25.350220 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.350191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr7tt\" (UniqueName: \"kubernetes.io/projected/493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b-kube-api-access-gr7tt\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg\" (UID: \"493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"
Sep 16 19:51:25.468345 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.468311    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"]
Sep 16 19:51:25.470430 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:51:25.470396    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3545dfcf_9e97_4a36_8e4b_d3c02e5e7a45.slice/crio-2153f920b14934b5360917d81cbc7ef9032fbf4771ef33f11d4256a4a4cb6a03 WatchSource:0}: Error finding container 2153f920b14934b5360917d81cbc7ef9032fbf4771ef33f11d4256a4a4cb6a03: Status 404 returned error can't find the container with id 2153f920b14934b5360917d81cbc7ef9032fbf4771ef33f11d4256a4a4cb6a03
Sep 16 19:51:25.472547 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.472526    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:51:25.518180 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.518148    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"
Sep 16 19:51:25.569382 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.569278    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" event={"ID":"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45","Type":"ContainerStarted","Data":"2153f920b14934b5360917d81cbc7ef9032fbf4771ef33f11d4256a4a4cb6a03"}
Sep 16 19:51:25.795837 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:25.795706    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"]
Sep 16 19:51:25.798681 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:51:25.798653    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493e1c4a_68a2_4e1c_a8a0_1529dbeeee7b.slice/crio-cdda3b5c1adbb95e5b8160b839e8daf2d3c5c762c8caffdd6a5ed6a6ac2df978 WatchSource:0}: Error finding container cdda3b5c1adbb95e5b8160b839e8daf2d3c5c762c8caffdd6a5ed6a6ac2df978: Status 404 returned error can't find the container with id cdda3b5c1adbb95e5b8160b839e8daf2d3c5c762c8caffdd6a5ed6a6ac2df978
Sep 16 19:51:26.577732 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:26.577678    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg" event={"ID":"493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b","Type":"ContainerStarted","Data":"bb727225044a03b378e4a52db8e05ea7a278293b218e6db9d10e3978a6a7b3c3"}
Sep 16 19:51:26.578288 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:26.577756    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg" event={"ID":"493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b","Type":"ContainerStarted","Data":"cdda3b5c1adbb95e5b8160b839e8daf2d3c5c762c8caffdd6a5ed6a6ac2df978"}
Sep 16 19:51:26.578288 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:26.577907    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"
Sep 16 19:51:26.595299 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:26.595183    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg" podStartSLOduration=1.595160843 podStartE2EDuration="1.595160843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:51:26.59488877 +0000 UTC m=+3944.425497317" watchObservedRunningTime="2026-09-16 19:51:26.595160843 +0000 UTC m=+3944.425769389"
Sep 16 19:51:27.585115 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:27.585072    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" event={"ID":"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45","Type":"ContainerStarted","Data":"3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073"}
Sep 16 19:51:27.585567 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:27.585279    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:28.591805 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:28.591761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" event={"ID":"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45","Type":"ContainerStarted","Data":"c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563"}
Sep 16 19:51:30.603538 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:30.603501    2582 generic.go:356] "Generic (PLEG): container finished" podID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerID="c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563" exitCode=0
Sep 16 19:51:30.604055 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:30.603594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" event={"ID":"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45","Type":"ContainerDied","Data":"c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563"}
Sep 16 19:51:31.612263 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:31.612216    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" event={"ID":"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45","Type":"ContainerStarted","Data":"051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4"}
Sep 16 19:51:31.649544 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:31.649406    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podStartSLOduration=3.892781664 podStartE2EDuration="7.64938162s" totalImagesPullingTime="1.497738757s" totalInitContainerRuntime="2.258861199s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:51:31.641271157 +0000 UTC m=+3949.471879703" watchObservedRunningTime="2026-09-16 19:51:31.64938162 +0000 UTC m=+3949.479990166"
Sep 16 19:51:31.709002 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:31.708957    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:32.833107 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:32.833070    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:35.180310 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:35.180276    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:35.180827 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:35.180338    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:35.181698 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:35.181670    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:51:38.434048 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:38.434004    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:45.180161 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:45.180105    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:51:45.192234 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:45.192203    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:51:55.119770 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:55.119724    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:55.180489 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:55.180448    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:51:56.505256 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:56.505226    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:57.586625 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:57.586593    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"
Sep 16 19:51:58.298529 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:51:58.298472    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:01.408853 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:01.408807    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:01.475832 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:01.475804    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:05.180323 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:05.180269    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:52:13.354285 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:13.354244    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:13.453504 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:13.453470    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:13.553743 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:13.553706    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:13.654785 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:13.654688    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:13.764801 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:13.764760    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:15.180398 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:15.180347    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:52:15.917168 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:15.917131    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:15.975511 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:15.975482    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:16.757230 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:16.757194    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:19.502957 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:19.502924    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:19.603261 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:19.603218    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:25.180375 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:25.180318    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:52:25.198850 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:25.198783    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:25.299344 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:25.299235    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:26.657054 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:26.657018    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:26.757426 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:26.757381    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:35.180679 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:35.180626    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:52:41.561817 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:41.561771    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:45.180590 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:45.180528    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:52:55.179695 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:52:55.179653    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:53:05.180598 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:53:05.180447    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:53:15.180122 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:53:15.180048    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:53:25.179993 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:53:25.179934    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8001/health\": dial tcp 10.134.0.84:8001: connect: connection refused"
Sep 16 19:53:35.189995 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:53:35.189961    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:53:35.201915 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:53:35.201887    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:53:45.046382 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:53:45.045842    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:45.048684 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:53:45.048652    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"]
Sep 16 19:53:45.061609 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:53:45.061556    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"]
Sep 16 19:53:46.246121 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:53:46.246081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:47.587056 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:53:47.587020    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg" podUID="493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:53:57.587022 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:53:57.586974    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg" podUID="493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:54:00.078834 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:00.078774    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg" podUID="493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b" containerName="main" containerID="cri-o://bb727225044a03b378e4a52db8e05ea7a278293b218e6db9d10e3978a6a7b3c3" gracePeriod=45
Sep 16 19:54:00.086725 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:00.086669    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main" containerID="cri-o://051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4" gracePeriod=45
Sep 16 19:54:00.805439 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:00.805402    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="llm-d-routing-sidecar" containerID="cri-o://3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073" gracePeriod=45
Sep 16 19:54:01.068316 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.068286    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:54:01.196623 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.196557    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-model-cache\") pod \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") "
Sep 16 19:54:01.196623 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.196626    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-tmp-dir\") pod \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") "
Sep 16 19:54:01.197187 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.196645    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-home\") pod \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") "
Sep 16 19:54:01.197187 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.196665    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4hfg\" (UniqueName: \"kubernetes.io/projected/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-kube-api-access-z4hfg\") pod \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") "
Sep 16 19:54:01.197187 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.196709    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-kserve-provision-location\") pod \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") "
Sep 16 19:54:01.197187 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.196771    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-tls-certs\") pod \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") "
Sep 16 19:54:01.197187 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.196844    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-cabundle-cert\") pod \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") "
Sep 16 19:54:01.197187 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.196894    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-model-cache" (OuterVolumeSpecName: "model-cache") pod "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" (UID: "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:01.197187 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.196881    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-dshm\") pod \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\" (UID: \"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45\") "
Sep 16 19:54:01.197563 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.197248    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.197563 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.197243    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" (UID: "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:54:01.197563 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.197453    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-home" (OuterVolumeSpecName: "home") pod "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" (UID: "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:01.199663 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.199618    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-kube-api-access-z4hfg" (OuterVolumeSpecName: "kube-api-access-z4hfg") pod "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" (UID: "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45"). InnerVolumeSpecName "kube-api-access-z4hfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:54:01.199815 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.199785    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" (UID: "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:54:01.199897 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.199870    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-dshm" (OuterVolumeSpecName: "dshm") pod "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" (UID: "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:01.213522 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.213467    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" (UID: "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:01.220968 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.220938    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" (UID: "3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:01.298285 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.298241    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-cabundle-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.298285 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.298277    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.298285 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.298285    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.298285 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.298295    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.298632 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.298304    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4hfg\" (UniqueName: \"kubernetes.io/projected/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-kube-api-access-z4hfg\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.298632 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.298314    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.298632 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.298322    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.349301 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.349256    2582 generic.go:356] "Generic (PLEG): container finished" podID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerID="051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4" exitCode=0
Sep 16 19:54:01.349301 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.349282    2582 generic.go:356] "Generic (PLEG): container finished" podID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerID="3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073" exitCode=0
Sep 16 19:54:01.349603 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.349318    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"
Sep 16 19:54:01.349603 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.349321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" event={"ID":"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45","Type":"ContainerDied","Data":"051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4"}
Sep 16 19:54:01.349603 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.349443    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" event={"ID":"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45","Type":"ContainerDied","Data":"3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073"}
Sep 16 19:54:01.349603 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.349460    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl" event={"ID":"3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45","Type":"ContainerDied","Data":"2153f920b14934b5360917d81cbc7ef9032fbf4771ef33f11d4256a4a4cb6a03"}
Sep 16 19:54:01.349603 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.349483    2582 scope.go:117] "RemoveContainer" containerID="051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4"
Sep 16 19:54:01.359470 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.359448    2582 scope.go:117] "RemoveContainer" containerID="c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563"
Sep 16 19:54:01.368410 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.368385    2582 scope.go:117] "RemoveContainer" containerID="3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073"
Sep 16 19:54:01.376243 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.376212    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"]
Sep 16 19:54:01.378088 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.378060    2582 scope.go:117] "RemoveContainer" containerID="051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4"
Sep 16 19:54:01.378631 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:54:01.378603    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4\": container with ID starting with 051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4 not found: ID does not exist" containerID="051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4"
Sep 16 19:54:01.378761 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.378639    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4"} err="failed to get container status \"051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4\": rpc error: code = NotFound desc = could not find container \"051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4\": container with ID starting with 051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4 not found: ID does not exist"
Sep 16 19:54:01.378761 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.378659    2582 scope.go:117] "RemoveContainer" containerID="c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563"
Sep 16 19:54:01.379038 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:54:01.379012    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563\": container with ID starting with c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563 not found: ID does not exist" containerID="c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563"
Sep 16 19:54:01.379118 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.379047    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563"} err="failed to get container status \"c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563\": rpc error: code = NotFound desc = could not find container \"c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563\": container with ID starting with c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563 not found: ID does not exist"
Sep 16 19:54:01.379118 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.379068    2582 scope.go:117] "RemoveContainer" containerID="3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073"
Sep 16 19:54:01.379361 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:54:01.379344    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073\": container with ID starting with 3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073 not found: ID does not exist" containerID="3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073"
Sep 16 19:54:01.379410 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.379366    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073"} err="failed to get container status \"3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073\": rpc error: code = NotFound desc = could not find container \"3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073\": container with ID starting with 3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073 not found: ID does not exist"
Sep 16 19:54:01.379410 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.379382    2582 scope.go:117] "RemoveContainer" containerID="051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4"
Sep 16 19:54:01.379699 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.379673    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4"} err="failed to get container status \"051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4\": rpc error: code = NotFound desc = could not find container \"051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4\": container with ID starting with 051b867961d378b8e22769710e26b9755aee86e8fa2c68e485c3f24a1c4dace4 not found: ID does not exist"
Sep 16 19:54:01.379775 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.379701    2582 scope.go:117] "RemoveContainer" containerID="c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563"
Sep 16 19:54:01.379984 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.379962    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563"} err="failed to get container status \"c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563\": rpc error: code = NotFound desc = could not find container \"c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563\": container with ID starting with c452e6a4ca47894512b07272e94215dd83d7dfa91934ca24ac092a1044ded563 not found: ID does not exist"
Sep 16 19:54:01.380072 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.379987    2582 scope.go:117] "RemoveContainer" containerID="3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073"
Sep 16 19:54:01.380289 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.380272    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073"} err="failed to get container status \"3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073\": rpc error: code = NotFound desc = could not find container \"3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073\": container with ID starting with 3e1a5f01041e0778059eff51d0a1c13148e4ba76a2f9ab38b61e2076aa724073 not found: ID does not exist"
Sep 16 19:54:01.380546 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:01.380529    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-6c4fc9cc48nrpjl"]
Sep 16 19:54:02.704519 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:02.704474    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" path="/var/lib/kubelet/pods/3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45/volumes"
Sep 16 19:54:07.587472 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:07.587428    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg" podUID="493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:54:07.587932 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:07.587569    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"
Sep 16 19:54:17.587458 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:17.587409    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg" podUID="493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:54:27.587449 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:27.587402    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg" podUID="493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:54:30.330010 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.329977    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"
Sep 16 19:54:30.466134 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.465990    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b-tls-certs\") pod \"493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b\" (UID: \"493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b\") "
Sep 16 19:54:30.466134 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.466064    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr7tt\" (UniqueName: \"kubernetes.io/projected/493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b-kube-api-access-gr7tt\") pod \"493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b\" (UID: \"493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b\") "
Sep 16 19:54:30.468810 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.468766    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b" (UID: "493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:54:30.468958 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.468893    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b-kube-api-access-gr7tt" (OuterVolumeSpecName: "kube-api-access-gr7tt") pod "493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b" (UID: "493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b"). InnerVolumeSpecName "kube-api-access-gr7tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:54:30.507641 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.507561    2582 generic.go:356] "Generic (PLEG): container finished" podID="493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b" containerID="bb727225044a03b378e4a52db8e05ea7a278293b218e6db9d10e3978a6a7b3c3" exitCode=0
Sep 16 19:54:30.507641 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.507643    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"
Sep 16 19:54:30.507970 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.507653    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg" event={"ID":"493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b","Type":"ContainerDied","Data":"bb727225044a03b378e4a52db8e05ea7a278293b218e6db9d10e3978a6a7b3c3"}
Sep 16 19:54:30.507970 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.507766    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg" event={"ID":"493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b","Type":"ContainerDied","Data":"cdda3b5c1adbb95e5b8160b839e8daf2d3c5c762c8caffdd6a5ed6a6ac2df978"}
Sep 16 19:54:30.507970 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.507794    2582 scope.go:117] "RemoveContainer" containerID="bb727225044a03b378e4a52db8e05ea7a278293b218e6db9d10e3978a6a7b3c3"
Sep 16 19:54:30.518485 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.518459    2582 scope.go:117] "RemoveContainer" containerID="bb727225044a03b378e4a52db8e05ea7a278293b218e6db9d10e3978a6a7b3c3"
Sep 16 19:54:30.518879 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:54:30.518846    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb727225044a03b378e4a52db8e05ea7a278293b218e6db9d10e3978a6a7b3c3\": container with ID starting with bb727225044a03b378e4a52db8e05ea7a278293b218e6db9d10e3978a6a7b3c3 not found: ID does not exist" containerID="bb727225044a03b378e4a52db8e05ea7a278293b218e6db9d10e3978a6a7b3c3"
Sep 16 19:54:30.518985 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.518891    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb727225044a03b378e4a52db8e05ea7a278293b218e6db9d10e3978a6a7b3c3"} err="failed to get container status \"bb727225044a03b378e4a52db8e05ea7a278293b218e6db9d10e3978a6a7b3c3\": rpc error: code = NotFound desc = could not find container \"bb727225044a03b378e4a52db8e05ea7a278293b218e6db9d10e3978a6a7b3c3\": container with ID starting with bb727225044a03b378e4a52db8e05ea7a278293b218e6db9d10e3978a6a7b3c3 not found: ID does not exist"
Sep 16 19:54:30.530619 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.530565    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"]
Sep 16 19:54:30.534477 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.534442    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelkqxg"]
Sep 16 19:54:30.567827 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.567772    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:54:30.567827 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.567824    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr7tt\" (UniqueName: \"kubernetes.io/projected/493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b-kube-api-access-gr7tt\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:54:30.704892 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:30.704848    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b" path="/var/lib/kubelet/pods/493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b/volumes"
Sep 16 19:54:35.247502 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:35.247399    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:35.344632 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:35.344586    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:41.504104 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.504063    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"]
Sep 16 19:54:41.504606 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.504567    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b" containerName="main"
Sep 16 19:54:41.504674 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.504608    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b" containerName="main"
Sep 16 19:54:41.504674 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.504638    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main"
Sep 16 19:54:41.504674 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.504647    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main"
Sep 16 19:54:41.504674 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.504669    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="llm-d-routing-sidecar"
Sep 16 19:54:41.504674 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.504675    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="llm-d-routing-sidecar"
Sep 16 19:54:41.504840 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.504684    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="storage-initializer"
Sep 16 19:54:41.504840 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.504690    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="storage-initializer"
Sep 16 19:54:41.504840 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.504770    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="493e1c4a-68a2-4e1c-a8a0-1529dbeeee7b" containerName="main"
Sep 16 19:54:41.504840 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.504779    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="llm-d-routing-sidecar"
Sep 16 19:54:41.504840 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.504786    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3545dfcf-9e97-4a36-8e4b-d3c02e5e7a45" containerName="main"
Sep 16 19:54:41.507429 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.507409    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.510988 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.510954    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 19:54:41.511164 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.510967    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 19:54:41.512416 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.512385    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-rs6n4\""
Sep 16 19:54:41.512416 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.512388    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 19:54:41.512672 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.512418    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:54:41.512672 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.512649    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-tp4mc\""
Sep 16 19:54:41.512798 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.512775    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 19:54:41.523290 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.523260    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"]
Sep 16 19:54:41.555997 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.555949    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:41.672993 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.672948    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b0d040-9a18-4b2f-a6fd-e791bede6daf-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.673227 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.673008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-dshm\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.673227 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.673032    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09b0d040-9a18-4b2f-a6fd-e791bede6daf-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.673227 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.673085    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-home\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.673227 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.673112    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.673227 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.673133    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rgz\" (UniqueName: \"kubernetes.io/projected/09b0d040-9a18-4b2f-a6fd-e791bede6daf-kube-api-access-w2rgz\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.673227 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.673155    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.673227 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.673175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.731081 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.731033    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"]
Sep 16 19:54:41.733932 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.733910    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"
Sep 16 19:54:41.737311 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.737283    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-58h7t\""
Sep 16 19:54:41.746839 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.746814    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"]
Sep 16 19:54:41.768559 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.768525    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:41.773708 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.773685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.773782 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.773719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rgz\" (UniqueName: \"kubernetes.io/projected/09b0d040-9a18-4b2f-a6fd-e791bede6daf-kube-api-access-w2rgz\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.773782 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.773744    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.773882 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.773778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.773882 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.773844    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b0d040-9a18-4b2f-a6fd-e791bede6daf-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.773970 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.773898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-dshm\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.773970 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.773924    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09b0d040-9a18-4b2f-a6fd-e791bede6daf-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.774059 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.773967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-home\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.774248 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.774221    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.774440 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.774252    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.774549 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.774287    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.774549 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.774461    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-home\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.774685 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.774666    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09b0d040-9a18-4b2f-a6fd-e791bede6daf-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.776546 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.776518    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-dshm\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.776700 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.776680    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b0d040-9a18-4b2f-a6fd-e791bede6daf-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.783050 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.783024    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rgz\" (UniqueName: \"kubernetes.io/projected/09b0d040-9a18-4b2f-a6fd-e791bede6daf-kube-api-access-w2rgz\") pod \"custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.819217 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.819176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:41.875273 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.875239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2929b0-3736-42c5-87b7-2d9cc84932da-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx\" (UID: \"5a2929b0-3736-42c5-87b7-2d9cc84932da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"
Sep 16 19:54:41.875422 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.875378    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmgr\" (UniqueName: \"kubernetes.io/projected/5a2929b0-3736-42c5-87b7-2d9cc84932da-kube-api-access-lxmgr\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx\" (UID: \"5a2929b0-3736-42c5-87b7-2d9cc84932da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"
Sep 16 19:54:41.976347 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.976305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2929b0-3736-42c5-87b7-2d9cc84932da-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx\" (UID: \"5a2929b0-3736-42c5-87b7-2d9cc84932da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"
Sep 16 19:54:41.976567 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.976375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmgr\" (UniqueName: \"kubernetes.io/projected/5a2929b0-3736-42c5-87b7-2d9cc84932da-kube-api-access-lxmgr\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx\" (UID: \"5a2929b0-3736-42c5-87b7-2d9cc84932da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"
Sep 16 19:54:41.979350 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.979293    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2929b0-3736-42c5-87b7-2d9cc84932da-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx\" (UID: \"5a2929b0-3736-42c5-87b7-2d9cc84932da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"
Sep 16 19:54:41.986124 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:41.986088    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmgr\" (UniqueName: \"kubernetes.io/projected/5a2929b0-3736-42c5-87b7-2d9cc84932da-kube-api-access-lxmgr\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx\" (UID: \"5a2929b0-3736-42c5-87b7-2d9cc84932da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"
Sep 16 19:54:42.026383 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:42.026268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:42.046004 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:42.045966    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"
Sep 16 19:54:42.104920 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:42.104350    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"]
Sep 16 19:54:42.110229 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:54:42.108525    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b0d040_9a18_4b2f_a6fd_e791bede6daf.slice/crio-1d83289b712961140b3be8e72c5fff3545511d5791abc7b1951fae38a102f857 WatchSource:0}: Error finding container 1d83289b712961140b3be8e72c5fff3545511d5791abc7b1951fae38a102f857: Status 404 returned error can't find the container with id 1d83289b712961140b3be8e72c5fff3545511d5791abc7b1951fae38a102f857
Sep 16 19:54:42.324045 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:42.324013    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"]
Sep 16 19:54:42.326232 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:54:42.326196    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2929b0_3736_42c5_87b7_2d9cc84932da.slice/crio-58ce6affa152c7fb6ebc1420bd1d6d5dff2eefd5489284046df1f0f40cdf2951 WatchSource:0}: Error finding container 58ce6affa152c7fb6ebc1420bd1d6d5dff2eefd5489284046df1f0f40cdf2951: Status 404 returned error can't find the container with id 58ce6affa152c7fb6ebc1420bd1d6d5dff2eefd5489284046df1f0f40cdf2951
Sep 16 19:54:42.565110 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:42.565065    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" event={"ID":"09b0d040-9a18-4b2f-a6fd-e791bede6daf","Type":"ContainerStarted","Data":"a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7"}
Sep 16 19:54:42.565685 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:42.565143    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" event={"ID":"09b0d040-9a18-4b2f-a6fd-e791bede6daf","Type":"ContainerStarted","Data":"1d83289b712961140b3be8e72c5fff3545511d5791abc7b1951fae38a102f857"}
Sep 16 19:54:42.565685 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:42.565269    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:42.566790 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:42.566751    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx" event={"ID":"5a2929b0-3736-42c5-87b7-2d9cc84932da","Type":"ContainerStarted","Data":"7b6ab47ef72687255240b7b7e626da15a6fd2bfab9e8ce52de08ecde31fb38e0"}
Sep 16 19:54:42.566935 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:42.566797    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx" event={"ID":"5a2929b0-3736-42c5-87b7-2d9cc84932da","Type":"ContainerStarted","Data":"58ce6affa152c7fb6ebc1420bd1d6d5dff2eefd5489284046df1f0f40cdf2951"}
Sep 16 19:54:42.566935 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:42.566864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"
Sep 16 19:54:42.606785 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:42.606724    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx" podStartSLOduration=1.606699877 podStartE2EDuration="1.606699877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:54:42.605136694 +0000 UTC m=+4140.435745252" watchObservedRunningTime="2026-09-16 19:54:42.606699877 +0000 UTC m=+4140.437308424"
Sep 16 19:54:43.572692 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:43.572651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" event={"ID":"09b0d040-9a18-4b2f-a6fd-e791bede6daf","Type":"ContainerStarted","Data":"7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277"}
Sep 16 19:54:45.585496 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:45.585452    2582 generic.go:356] "Generic (PLEG): container finished" podID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerID="7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277" exitCode=0
Sep 16 19:54:45.585496 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:45.585496    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" event={"ID":"09b0d040-9a18-4b2f-a6fd-e791bede6daf","Type":"ContainerDied","Data":"7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277"}
Sep 16 19:54:46.593116 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:46.593069    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" event={"ID":"09b0d040-9a18-4b2f-a6fd-e791bede6daf","Type":"ContainerStarted","Data":"09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5"}
Sep 16 19:54:46.619648 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:46.619562    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podStartSLOduration=3.227140108 podStartE2EDuration="5.619538211s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.392398103s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:54:46.618142138 +0000 UTC m=+4144.448750689" watchObservedRunningTime="2026-09-16 19:54:46.619538211 +0000 UTC m=+4144.450146757"
Sep 16 19:54:47.521374 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:47.521341    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:47.582885 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:47.582846    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:51.305702 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:51.305669    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:51.820394 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:51.820355    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:51.820622 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:51.820431    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:54:51.821664 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:51.821632    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:54:58.925009 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:54:58.924971    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:00.628149 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:00.628108    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:01.819872 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:01.819815    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:55:01.832993 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:01.832951    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:55:02.464405 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:02.464368    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:02.518373 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:02.518332    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:11.819906 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:11.819850    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:55:13.575017 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:13.574980    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"
Sep 16 19:55:20.043962 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:20.043927    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:20.142457 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:20.142415    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:21.820376 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:21.820309    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:55:25.497999 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:25.497895    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:25.557882 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:25.557847    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:26.204385 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:26.204350    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:28.952567 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:28.952525    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:29.201481 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:29.201454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:31.820184 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:31.820135    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:55:32.126191 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:32.126093    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:32.199249 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:32.199201    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:32.653036 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:32.652988    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:32.714834 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:32.714795    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:40.641635 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:40.641589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:41.820039 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:41.819964    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:55:42.735061 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:42.735003    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:51.820285 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:55:51.820229    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:56:01.819765 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:56:01.819706    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:56:11.819979 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:56:11.819923    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:56:21.820733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:56:21.820668    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:56:31.819873 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:56:31.819806    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:56:41.820152 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:56:41.820093    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8001/health\": dial tcp 10.134.0.86:8001: connect: connection refused"
Sep 16 19:56:51.829457 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:56:51.829413    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:56:51.841865 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:56:51.841830    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:57:04.252997 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:04.252937    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:04.254251 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:04.254223    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"]
Sep 16 19:57:04.261025 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:04.260992    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"]
Sep 16 19:57:04.328420 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:04.328378    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:13.575556 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:13.575502    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx" podUID="5a2929b0-3736-42c5-87b7-2d9cc84932da" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:57:19.276296 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:19.276255    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx" podUID="5a2929b0-3736-42c5-87b7-2d9cc84932da" containerName="main" containerID="cri-o://7b6ab47ef72687255240b7b7e626da15a6fd2bfab9e8ce52de08ecde31fb38e0" gracePeriod=45
Sep 16 19:57:19.285621 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:19.285554    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main" containerID="cri-o://09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5" gracePeriod=45
Sep 16 19:57:19.826210 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:19.826183    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:19.902770 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:19.902718    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="llm-d-routing-sidecar" containerID="cri-o://a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7" gracePeriod=45
Sep 16 19:57:20.147452 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.147424    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:57:20.238722 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.238683    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-model-cache\") pod \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") "
Sep 16 19:57:20.238722 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.238721    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-kserve-provision-location\") pod \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") "
Sep 16 19:57:20.238968 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.238753    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2rgz\" (UniqueName: \"kubernetes.io/projected/09b0d040-9a18-4b2f-a6fd-e791bede6daf-kube-api-access-w2rgz\") pod \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") "
Sep 16 19:57:20.238968 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.238800    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-tmp-dir\") pod \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") "
Sep 16 19:57:20.238968 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.238828    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b0d040-9a18-4b2f-a6fd-e791bede6daf-tls-certs\") pod \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") "
Sep 16 19:57:20.238968 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.238846    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09b0d040-9a18-4b2f-a6fd-e791bede6daf-cabundle-cert\") pod \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") "
Sep 16 19:57:20.238968 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.238921    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-dshm\") pod \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") "
Sep 16 19:57:20.239223 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.238971    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-home\") pod \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\" (UID: \"09b0d040-9a18-4b2f-a6fd-e791bede6daf\") "
Sep 16 19:57:20.239223 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.239027    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-model-cache" (OuterVolumeSpecName: "model-cache") pod "09b0d040-9a18-4b2f-a6fd-e791bede6daf" (UID: "09b0d040-9a18-4b2f-a6fd-e791bede6daf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:20.239223 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.239200    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b0d040-9a18-4b2f-a6fd-e791bede6daf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09b0d040-9a18-4b2f-a6fd-e791bede6daf" (UID: "09b0d040-9a18-4b2f-a6fd-e791bede6daf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:57:20.239403 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.239245    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.239803 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.239781    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-home" (OuterVolumeSpecName: "home") pod "09b0d040-9a18-4b2f-a6fd-e791bede6daf" (UID: "09b0d040-9a18-4b2f-a6fd-e791bede6daf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:20.241394 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.241366    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b0d040-9a18-4b2f-a6fd-e791bede6daf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09b0d040-9a18-4b2f-a6fd-e791bede6daf" (UID: "09b0d040-9a18-4b2f-a6fd-e791bede6daf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:57:20.241518 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.241493    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b0d040-9a18-4b2f-a6fd-e791bede6daf-kube-api-access-w2rgz" (OuterVolumeSpecName: "kube-api-access-w2rgz") pod "09b0d040-9a18-4b2f-a6fd-e791bede6daf" (UID: "09b0d040-9a18-4b2f-a6fd-e791bede6daf"). InnerVolumeSpecName "kube-api-access-w2rgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:20.241622 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.241604    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-dshm" (OuterVolumeSpecName: "dshm") pod "09b0d040-9a18-4b2f-a6fd-e791bede6daf" (UID: "09b0d040-9a18-4b2f-a6fd-e791bede6daf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:20.251535 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.251508    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09b0d040-9a18-4b2f-a6fd-e791bede6daf" (UID: "09b0d040-9a18-4b2f-a6fd-e791bede6daf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:20.263673 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.263648    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09b0d040-9a18-4b2f-a6fd-e791bede6daf" (UID: "09b0d040-9a18-4b2f-a6fd-e791bede6daf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:20.340230 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.340142    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.340230 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.340176    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2rgz\" (UniqueName: \"kubernetes.io/projected/09b0d040-9a18-4b2f-a6fd-e791bede6daf-kube-api-access-w2rgz\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.340230 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.340186    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.340230 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.340196    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09b0d040-9a18-4b2f-a6fd-e791bede6daf-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.340230 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.340205    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09b0d040-9a18-4b2f-a6fd-e791bede6daf-cabundle-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.340230 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.340212    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.340230 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.340219    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09b0d040-9a18-4b2f-a6fd-e791bede6daf-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.351733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.351695    2582 generic.go:356] "Generic (PLEG): container finished" podID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerID="09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5" exitCode=0
Sep 16 19:57:20.351733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.351725    2582 generic.go:356] "Generic (PLEG): container finished" podID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerID="a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7" exitCode=0
Sep 16 19:57:20.351935 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.351753    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" event={"ID":"09b0d040-9a18-4b2f-a6fd-e791bede6daf","Type":"ContainerDied","Data":"09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5"}
Sep 16 19:57:20.351935 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.351763    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"
Sep 16 19:57:20.351935 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.351830    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" event={"ID":"09b0d040-9a18-4b2f-a6fd-e791bede6daf","Type":"ContainerDied","Data":"a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7"}
Sep 16 19:57:20.351935 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.351850    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg" event={"ID":"09b0d040-9a18-4b2f-a6fd-e791bede6daf","Type":"ContainerDied","Data":"1d83289b712961140b3be8e72c5fff3545511d5791abc7b1951fae38a102f857"}
Sep 16 19:57:20.351935 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.351869    2582 scope.go:117] "RemoveContainer" containerID="09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5"
Sep 16 19:57:20.361064 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.361047    2582 scope.go:117] "RemoveContainer" containerID="7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277"
Sep 16 19:57:20.368546 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.368524    2582 scope.go:117] "RemoveContainer" containerID="a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7"
Sep 16 19:57:20.375552 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.375525    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"]
Sep 16 19:57:20.376207 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.376183    2582 scope.go:117] "RemoveContainer" containerID="09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5"
Sep 16 19:57:20.376512 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:57:20.376492    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5\": container with ID starting with 09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5 not found: ID does not exist" containerID="09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5"
Sep 16 19:57:20.376610 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.376524    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5"} err="failed to get container status \"09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5\": rpc error: code = NotFound desc = could not find container \"09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5\": container with ID starting with 09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5 not found: ID does not exist"
Sep 16 19:57:20.376610 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.376551    2582 scope.go:117] "RemoveContainer" containerID="7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277"
Sep 16 19:57:20.377417 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:57:20.377388    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277\": container with ID starting with 7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277 not found: ID does not exist" containerID="7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277"
Sep 16 19:57:20.377498 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.377426    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277"} err="failed to get container status \"7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277\": rpc error: code = NotFound desc = could not find container \"7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277\": container with ID starting with 7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277 not found: ID does not exist"
Sep 16 19:57:20.377498 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.377451    2582 scope.go:117] "RemoveContainer" containerID="a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7"
Sep 16 19:57:20.377861 ip-10-0-142-173 kubenswrapper[2582]: E0916 19:57:20.377833    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7\": container with ID starting with a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7 not found: ID does not exist" containerID="a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7"
Sep 16 19:57:20.377953 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.377870    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7"} err="failed to get container status \"a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7\": rpc error: code = NotFound desc = could not find container \"a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7\": container with ID starting with a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7 not found: ID does not exist"
Sep 16 19:57:20.377953 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.377894    2582 scope.go:117] "RemoveContainer" containerID="09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5"
Sep 16 19:57:20.378191 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.378167    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5"} err="failed to get container status \"09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5\": rpc error: code = NotFound desc = could not find container \"09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5\": container with ID starting with 09120e285904b705547fc42d69189508353003be4c9773537f330ae8202749a5 not found: ID does not exist"
Sep 16 19:57:20.378191 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.378189    2582 scope.go:117] "RemoveContainer" containerID="7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277"
Sep 16 19:57:20.378415 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.378395    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277"} err="failed to get container status \"7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277\": rpc error: code = NotFound desc = could not find container \"7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277\": container with ID starting with 7142b43dcac4e47d90d29cc835f3d486c2071718b87cec250aa7bdbfa2fd5277 not found: ID does not exist"
Sep 16 19:57:20.378489 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.378418    2582 scope.go:117] "RemoveContainer" containerID="a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7"
Sep 16 19:57:20.378715 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.378695    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7"} err="failed to get container status \"a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7\": rpc error: code = NotFound desc = could not find container \"a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7\": container with ID starting with a2be929f55d1ad831f7f9ea7bd76a4204faaf291b18b7427780cc2e3d9650cb7 not found: ID does not exist"
Sep 16 19:57:20.379129 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.379111    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-5ccbcf7b4f-98zcg"]
Sep 16 19:57:20.703697 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:20.703613    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" path="/var/lib/kubelet/pods/09b0d040-9a18-4b2f-a6fd-e791bede6daf/volumes"
Sep 16 19:57:23.574113 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:23.574070    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx" podUID="5a2929b0-3736-42c5-87b7-2d9cc84932da" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:57:33.575034 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:33.574940    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx" podUID="5a2929b0-3736-42c5-87b7-2d9cc84932da" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:57:33.575433 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:33.575061    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"
Sep 16 19:57:34.567554 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:34.567516    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:34.634085 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:34.634036    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:43.575034 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:43.574993    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx" podUID="5a2929b0-3736-42c5-87b7-2d9cc84932da" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:57:49.481367 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:49.481334    2582 generic.go:356] "Generic (PLEG): container finished" podID="5a2929b0-3736-42c5-87b7-2d9cc84932da" containerID="7b6ab47ef72687255240b7b7e626da15a6fd2bfab9e8ce52de08ecde31fb38e0" exitCode=0
Sep 16 19:57:49.481751 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:49.481404    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx" event={"ID":"5a2929b0-3736-42c5-87b7-2d9cc84932da","Type":"ContainerDied","Data":"7b6ab47ef72687255240b7b7e626da15a6fd2bfab9e8ce52de08ecde31fb38e0"}
Sep 16 19:57:49.528811 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:49.528788    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"
Sep 16 19:57:49.614942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:49.614841    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxmgr\" (UniqueName: \"kubernetes.io/projected/5a2929b0-3736-42c5-87b7-2d9cc84932da-kube-api-access-lxmgr\") pod \"5a2929b0-3736-42c5-87b7-2d9cc84932da\" (UID: \"5a2929b0-3736-42c5-87b7-2d9cc84932da\") "
Sep 16 19:57:49.614942 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:49.614903    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2929b0-3736-42c5-87b7-2d9cc84932da-tls-certs\") pod \"5a2929b0-3736-42c5-87b7-2d9cc84932da\" (UID: \"5a2929b0-3736-42c5-87b7-2d9cc84932da\") "
Sep 16 19:57:49.617184 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:49.617158    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a2929b0-3736-42c5-87b7-2d9cc84932da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a2929b0-3736-42c5-87b7-2d9cc84932da" (UID: "5a2929b0-3736-42c5-87b7-2d9cc84932da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:57:49.617256 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:49.617222    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2929b0-3736-42c5-87b7-2d9cc84932da-kube-api-access-lxmgr" (OuterVolumeSpecName: "kube-api-access-lxmgr") pod "5a2929b0-3736-42c5-87b7-2d9cc84932da" (UID: "5a2929b0-3736-42c5-87b7-2d9cc84932da"). InnerVolumeSpecName "kube-api-access-lxmgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:49.715802 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:49.715766    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxmgr\" (UniqueName: \"kubernetes.io/projected/5a2929b0-3736-42c5-87b7-2d9cc84932da-kube-api-access-lxmgr\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:57:49.715802 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:49.715795    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2929b0-3736-42c5-87b7-2d9cc84932da-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 19:57:50.493494 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:50.493456    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx" event={"ID":"5a2929b0-3736-42c5-87b7-2d9cc84932da","Type":"ContainerDied","Data":"58ce6affa152c7fb6ebc1420bd1d6d5dff2eefd5489284046df1f0f40cdf2951"}
Sep 16 19:57:50.493952 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:50.493502    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"
Sep 16 19:57:50.493952 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:50.493545    2582 scope.go:117] "RemoveContainer" containerID="7b6ab47ef72687255240b7b7e626da15a6fd2bfab9e8ce52de08ecde31fb38e0"
Sep 16 19:57:50.516498 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:50.516475    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"]
Sep 16 19:57:50.517851 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:50.517831    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48btggx"]
Sep 16 19:57:50.703232 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:50.703197    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a2929b0-3736-42c5-87b7-2d9cc84932da" path="/var/lib/kubelet/pods/5a2929b0-3736-42c5-87b7-2d9cc84932da/volumes"
Sep 16 19:57:54.292519 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.292485    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:54.401111 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.401079    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:54.491340 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.491307    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"]
Sep 16 19:57:54.491778 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.491755    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="llm-d-routing-sidecar"
Sep 16 19:57:54.491778 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.491776    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="llm-d-routing-sidecar"
Sep 16 19:57:54.491954 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.491794    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="storage-initializer"
Sep 16 19:57:54.491954 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.491800    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="storage-initializer"
Sep 16 19:57:54.491954 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.491806    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main"
Sep 16 19:57:54.491954 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.491812    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main"
Sep 16 19:57:54.491954 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.491828    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2929b0-3736-42c5-87b7-2d9cc84932da" containerName="main"
Sep 16 19:57:54.491954 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.491834    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2929b0-3736-42c5-87b7-2d9cc84932da" containerName="main"
Sep 16 19:57:54.491954 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.491886    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a2929b0-3736-42c5-87b7-2d9cc84932da" containerName="main"
Sep 16 19:57:54.491954 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.491894    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="main"
Sep 16 19:57:54.491954 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.491900    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="09b0d040-9a18-4b2f-a6fd-e791bede6daf" containerName="llm-d-routing-sidecar"
Sep 16 19:57:54.496514 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.496490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.499831 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.499810    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:57:54.500020 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.499998    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-hsxrt\""
Sep 16 19:57:54.500294 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.500279    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 19:57:54.501116 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.501100    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 19:57:54.508448 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.508422    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"]
Sep 16 19:57:54.530058 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.530035    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:54.550190 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.550098    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-workload-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.550190 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.550144    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa1270e-8c20-41ba-ba08-2c939402ef54-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.550373 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.550202    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-token\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.550373 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.550223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-workload-certs\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.550373 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.550244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-podinfo\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.550373 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.550288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-envoy\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.550373 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.550325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7f4\" (UniqueName: \"kubernetes.io/projected/4fa1270e-8c20-41ba-ba08-2c939402ef54-kube-api-access-vd7f4\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.550541 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.550394    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-data\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.550541 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.550429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-credential-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.611642 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.611610    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:54.651149 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.651118    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-data\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.651149 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.651152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-credential-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.651367 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.651174    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-workload-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.651367 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.651201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa1270e-8c20-41ba-ba08-2c939402ef54-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.651367 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.651240    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-token\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.651367 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.651281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-workload-certs\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.651367 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.651302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-podinfo\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.651367 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.651332    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-envoy\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.651709 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.651375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7f4\" (UniqueName: \"kubernetes.io/projected/4fa1270e-8c20-41ba-ba08-2c939402ef54-kube-api-access-vd7f4\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.651709 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.651595    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-data\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.651709 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.651648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-credential-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.651857 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.651750    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-workload-socket\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.651857 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.651779    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-workload-certs\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.652051 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.652033    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa1270e-8c20-41ba-ba08-2c939402ef54-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.653989 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.653966    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-envoy\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.654291 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.654274    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-podinfo\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.663120 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.663097    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-token\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.663212 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.663201    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7f4\" (UniqueName: \"kubernetes.io/projected/4fa1270e-8c20-41ba-ba08-2c939402ef54-kube-api-access-vd7f4\") pod \"router-gateway-2-openshift-default-5858dff66d-fpz2m\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:54.807295 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:54.807196    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:55.087460 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:55.087431    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"]
Sep 16 19:57:55.089482 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:57:55.089452    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa1270e_8c20_41ba_ba08_2c939402ef54.slice/crio-a032bc892f66f49630179f5cf7a7c909cc0b314bdfac5a8bbd0347e987793117 WatchSource:0}: Error finding container a032bc892f66f49630179f5cf7a7c909cc0b314bdfac5a8bbd0347e987793117: Status 404 returned error can't find the container with id a032bc892f66f49630179f5cf7a7c909cc0b314bdfac5a8bbd0347e987793117
Sep 16 19:57:55.091494 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:55.091462    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:57:55.094003 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:55.092047    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:57:55.094003 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:55.092219    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 19:57:55.523970 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:55.523933    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m" event={"ID":"4fa1270e-8c20-41ba-ba08-2c939402ef54","Type":"ContainerStarted","Data":"e85d312a77462725d18361c30a350b7fc301e6f5af8b996dc7399ddc77c25030"}
Sep 16 19:57:55.524462 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:55.524199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m" event={"ID":"4fa1270e-8c20-41ba-ba08-2c939402ef54","Type":"ContainerStarted","Data":"a032bc892f66f49630179f5cf7a7c909cc0b314bdfac5a8bbd0347e987793117"}
Sep 16 19:57:55.547969 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:55.547912    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m" podStartSLOduration=1.547894571 podStartE2EDuration="1.547894571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:55.547107202 +0000 UTC m=+4333.377715748" watchObservedRunningTime="2026-09-16 19:57:55.547894571 +0000 UTC m=+4333.378503115"
Sep 16 19:57:55.808273 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:55.808164    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:57:56.809726 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:56.809670    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m" podUID="4fa1270e-8c20-41ba-ba08-2c939402ef54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Sep 16 19:57:57.808352 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:57.808302    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m" podUID="4fa1270e-8c20-41ba-ba08-2c939402ef54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Sep 16 19:57:58.440004 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:58.439972    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:58.807792 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:58.807742    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m" podUID="4fa1270e-8c20-41ba-ba08-2c939402ef54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Sep 16 19:57:59.808256 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:57:59.808205    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m" podUID="4fa1270e-8c20-41ba-ba08-2c939402ef54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Sep 16 19:58:00.719600 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:00.719542    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:00.808747 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:00.808688    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m" podUID="4fa1270e-8c20-41ba-ba08-2c939402ef54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Sep 16 19:58:01.811926 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:01.811893    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:58:01.812314 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:01.812176    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:58:01.813061 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:01.813044    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 19:58:05.761117 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.761080    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"]
Sep 16 19:58:05.764392 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.764375    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.768311 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.768286    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 19:58:05.768630 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.768608    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:58:05.768688 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.768651    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 19:58:05.768743 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.768705    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-sq4xj\""
Sep 16 19:58:05.768874 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.768861    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-wm6n4\""
Sep 16 19:58:05.780838 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.780808    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"]
Sep 16 19:58:05.829217 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.829183    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-home\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.829384 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.829228    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48121725-2569-4e97-b2e1-ca826ca44fe0-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.829384 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.829250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4nqc\" (UniqueName: \"kubernetes.io/projected/48121725-2569-4e97-b2e1-ca826ca44fe0-kube-api-access-x4nqc\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.829384 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.829343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.829384 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.829378    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-model-cache\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.829550 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.829412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-dshm\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.829550 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.829426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48121725-2569-4e97-b2e1-ca826ca44fe0-tls-certs\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.829550 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.829442    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.845451 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.845396    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:05.930071 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.930038    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-model-cache\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.930250 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.930088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-dshm\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.930250 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.930114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48121725-2569-4e97-b2e1-ca826ca44fe0-tls-certs\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.930250 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.930231    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.930419 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.930277    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-home\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.930419 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.930333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48121725-2569-4e97-b2e1-ca826ca44fe0-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.930419 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.930372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4nqc\" (UniqueName: \"kubernetes.io/projected/48121725-2569-4e97-b2e1-ca826ca44fe0-kube-api-access-x4nqc\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.930606 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.930451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.930606 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.930474    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-model-cache\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.930720 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.930649    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.930720 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.930691    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-home\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.930827 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.930809    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.931039 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.931020    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48121725-2569-4e97-b2e1-ca826ca44fe0-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.932733 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.932713    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48121725-2569-4e97-b2e1-ca826ca44fe0-tls-certs\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.932822 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.932793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-dshm\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:05.941075 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:05.941042    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4nqc\" (UniqueName: \"kubernetes.io/projected/48121725-2569-4e97-b2e1-ca826ca44fe0-kube-api-access-x4nqc\") pod \"router-with-refs-pd-test-kserve-6fc447d467-8n289\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:06.056749 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.056715    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"]
Sep 16 19:58:06.061044 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.061023    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"
Sep 16 19:58:06.063787 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.063767    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-svqdj\""
Sep 16 19:58:06.069251 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.069222    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"]
Sep 16 19:58:06.074891 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.074867    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:06.132371 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.132338    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sflg\" (UniqueName: \"kubernetes.io/projected/11cb76bb-0f84-48fc-ac45-177e2e9db0b2-kube-api-access-2sflg\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz\" (UID: \"11cb76bb-0f84-48fc-ac45-177e2e9db0b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"
Sep 16 19:58:06.132476 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.132404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cb76bb-0f84-48fc-ac45-177e2e9db0b2-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz\" (UID: \"11cb76bb-0f84-48fc-ac45-177e2e9db0b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"
Sep 16 19:58:06.233914 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.233874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sflg\" (UniqueName: \"kubernetes.io/projected/11cb76bb-0f84-48fc-ac45-177e2e9db0b2-kube-api-access-2sflg\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz\" (UID: \"11cb76bb-0f84-48fc-ac45-177e2e9db0b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"
Sep 16 19:58:06.233914 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.233918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cb76bb-0f84-48fc-ac45-177e2e9db0b2-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz\" (UID: \"11cb76bb-0f84-48fc-ac45-177e2e9db0b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"
Sep 16 19:58:06.236656 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.236636    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cb76bb-0f84-48fc-ac45-177e2e9db0b2-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz\" (UID: \"11cb76bb-0f84-48fc-ac45-177e2e9db0b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"
Sep 16 19:58:06.243704 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.243681    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sflg\" (UniqueName: \"kubernetes.io/projected/11cb76bb-0f84-48fc-ac45-177e2e9db0b2-kube-api-access-2sflg\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz\" (UID: \"11cb76bb-0f84-48fc-ac45-177e2e9db0b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"
Sep 16 19:58:06.290990 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.290962    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:06.356484 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.356459    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"]
Sep 16 19:58:06.358469 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:58:06.358440    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48121725_2569_4e97_b2e1_ca826ca44fe0.slice/crio-3609f3761fd9a5ef5721037c2c224836261f13dd4404a894cf2d80658da920c8 WatchSource:0}: Error finding container 3609f3761fd9a5ef5721037c2c224836261f13dd4404a894cf2d80658da920c8: Status 404 returned error can't find the container with id 3609f3761fd9a5ef5721037c2c224836261f13dd4404a894cf2d80658da920c8
Sep 16 19:58:06.372812 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.372789    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"
Sep 16 19:58:06.403148 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.403115    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:06.574818 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.574779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" event={"ID":"48121725-2569-4e97-b2e1-ca826ca44fe0","Type":"ContainerStarted","Data":"ca00cf6cc31a6387f69d4ede6fcdab8d4098de96f59745cb91449b99620f9350"}
Sep 16 19:58:06.574996 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.574842    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:06.574996 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.574867    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" event={"ID":"48121725-2569-4e97-b2e1-ca826ca44fe0","Type":"ContainerStarted","Data":"3609f3761fd9a5ef5721037c2c224836261f13dd4404a894cf2d80658da920c8"}
Sep 16 19:58:06.654244 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:06.654221    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"]
Sep 16 19:58:06.656691 ip-10-0-142-173 kubenswrapper[2582]: W0916 19:58:06.656648    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cb76bb_0f84_48fc_ac45_177e2e9db0b2.slice/crio-f9e54d665b7dbb533b9019f44a5e495c3110e2fc7d694e772d9c359f40fe9089 WatchSource:0}: Error finding container f9e54d665b7dbb533b9019f44a5e495c3110e2fc7d694e772d9c359f40fe9089: Status 404 returned error can't find the container with id f9e54d665b7dbb533b9019f44a5e495c3110e2fc7d694e772d9c359f40fe9089
Sep 16 19:58:07.149641 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:07.149565    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:07.216996 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:07.216956    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:07.580222 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:07.580178    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz" event={"ID":"11cb76bb-0f84-48fc-ac45-177e2e9db0b2","Type":"ContainerStarted","Data":"409b78144abcb9851259096ea50cfdd9e6202210f5b0023bed9c67961a0a0d32"}
Sep 16 19:58:07.580404 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:07.580243    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"
Sep 16 19:58:07.580404 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:07.580269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz" event={"ID":"11cb76bb-0f84-48fc-ac45-177e2e9db0b2","Type":"ContainerStarted","Data":"f9e54d665b7dbb533b9019f44a5e495c3110e2fc7d694e772d9c359f40fe9089"}
Sep 16 19:58:07.581908 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:07.581887    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" event={"ID":"48121725-2569-4e97-b2e1-ca826ca44fe0","Type":"ContainerStarted","Data":"09764b2d520bb503997f606a629bdb532985e34639f1f9c0328ec32247bcb29f"}
Sep 16 19:58:07.597357 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:07.597308    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz" podStartSLOduration=1.597293369 podStartE2EDuration="1.597293369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:58:07.596315829 +0000 UTC m=+4345.426924375" watchObservedRunningTime="2026-09-16 19:58:07.597293369 +0000 UTC m=+4345.427901914"
Sep 16 19:58:09.591936 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:09.591902    2582 generic.go:356] "Generic (PLEG): container finished" podID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerID="09764b2d520bb503997f606a629bdb532985e34639f1f9c0328ec32247bcb29f" exitCode=0
Sep 16 19:58:09.592303 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:09.591944    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" event={"ID":"48121725-2569-4e97-b2e1-ca826ca44fe0","Type":"ContainerDied","Data":"09764b2d520bb503997f606a629bdb532985e34639f1f9c0328ec32247bcb29f"}
Sep 16 19:58:10.598848 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:10.598806    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" event={"ID":"48121725-2569-4e97-b2e1-ca826ca44fe0","Type":"ContainerStarted","Data":"d5dc5474adf18918074878f6f020454360f4fe31d43b66bbbd695f273a54a052"}
Sep 16 19:58:10.627685 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:10.627638    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" podStartSLOduration=3.424074729 podStartE2EDuration="5.627616566s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.203541837s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:58:10.624512933 +0000 UTC m=+4348.455121481" watchObservedRunningTime="2026-09-16 19:58:10.627616566 +0000 UTC m=+4348.458225113"
Sep 16 19:58:16.075621 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:16.075554    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:16.076163 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:16.075665    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:16.077023 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:16.076974    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8001/health\": dial tcp 10.134.0.89:8001: connect: connection refused"
Sep 16 19:58:16.088679 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:16.088656    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 19:58:24.676799 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:24.676760    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:24.771894 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:24.771854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:26.076032 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:26.075972    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8001/health\": dial tcp 10.134.0.89:8001: connect: connection refused"
Sep 16 19:58:30.620541 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:30.620506    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:36.075919 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:36.075870    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8001/health\": dial tcp 10.134.0.89:8001: connect: connection refused"
Sep 16 19:58:36.478477 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:36.478402    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:36.579655 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:36.579626    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:38.587926 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:38.587892    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"
Sep 16 19:58:43.648478 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:43.648447    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:43.944291 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:43.944207    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:46.075979 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:46.075935    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8001/health\": dial tcp 10.134.0.89:8001: connect: connection refused"
Sep 16 19:58:46.266275 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:46.266233    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:56.075308 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:58:56.075253    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8001/health\": dial tcp 10.134.0.89:8001: connect: connection refused"
Sep 16 19:59:06.075723 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:59:06.075596    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8001/health\": dial tcp 10.134.0.89:8001: connect: connection refused"
Sep 16 19:59:06.515369 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:59:06.515288    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 19:59:16.076228 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:59:16.076177    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8001/health\": dial tcp 10.134.0.89:8001: connect: connection refused"
Sep 16 19:59:26.076177 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:59:26.076133    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8001/health\": dial tcp 10.134.0.89:8001: connect: connection refused"
Sep 16 19:59:36.075770 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:59:36.075723    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8001/health\": dial tcp 10.134.0.89:8001: connect: connection refused"
Sep 16 19:59:46.075641 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:59:46.075587    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8001/health\": dial tcp 10.134.0.89:8001: connect: connection refused"
Sep 16 19:59:56.075837 ip-10-0-142-173 kubenswrapper[2582]: I0916 19:59:56.075784    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8001/health\": dial tcp 10.134.0.89:8001: connect: connection refused"
Sep 16 20:00:06.085431 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:06.085393    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 20:00:06.104096 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:06.104064    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 20:00:18.396541 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:18.396501    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:18.497565 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:18.497521    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:19.650439 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:19.650379    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:19.655074 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:19.655040    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"]
Sep 16 20:00:19.663085 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:19.663056    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"]
Sep 16 20:00:19.725982 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:19.725949    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:28.588008 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:28.587960    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz" podUID="11cb76bb-0f84-48fc-ac45-177e2e9db0b2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:34.677436 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:34.677329    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz" podUID="11cb76bb-0f84-48fc-ac45-177e2e9db0b2" containerName="main" containerID="cri-o://409b78144abcb9851259096ea50cfdd9e6202210f5b0023bed9c67961a0a0d32" gracePeriod=45
Sep 16 20:00:34.687966 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:34.687922    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main" containerID="cri-o://d5dc5474adf18918074878f6f020454360f4fe31d43b66bbbd695f273a54a052" gracePeriod=45
Sep 16 20:00:35.305423 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.305392    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="llm-d-routing-sidecar" containerID="cri-o://ca00cf6cc31a6387f69d4ede6fcdab8d4098de96f59745cb91449b99620f9350" gracePeriod=45
Sep 16 20:00:35.310008 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.309983    2582 generic.go:356] "Generic (PLEG): container finished" podID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerID="d5dc5474adf18918074878f6f020454360f4fe31d43b66bbbd695f273a54a052" exitCode=0
Sep 16 20:00:35.310117 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.310054    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" event={"ID":"48121725-2569-4e97-b2e1-ca826ca44fe0","Type":"ContainerDied","Data":"d5dc5474adf18918074878f6f020454360f4fe31d43b66bbbd695f273a54a052"}
Sep 16 20:00:35.558770 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.558743    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 20:00:35.667925 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.667889    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48121725-2569-4e97-b2e1-ca826ca44fe0-cabundle-cert\") pod \"48121725-2569-4e97-b2e1-ca826ca44fe0\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") "
Sep 16 20:00:35.668109 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.667945    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-model-cache\") pod \"48121725-2569-4e97-b2e1-ca826ca44fe0\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") "
Sep 16 20:00:35.668109 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.667966    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-kserve-provision-location\") pod \"48121725-2569-4e97-b2e1-ca826ca44fe0\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") "
Sep 16 20:00:35.668109 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.667991    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-tmp-dir\") pod \"48121725-2569-4e97-b2e1-ca826ca44fe0\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") "
Sep 16 20:00:35.668109 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.668031    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-home\") pod \"48121725-2569-4e97-b2e1-ca826ca44fe0\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") "
Sep 16 20:00:35.668109 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.668046    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-dshm\") pod \"48121725-2569-4e97-b2e1-ca826ca44fe0\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") "
Sep 16 20:00:35.668109 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.668083    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4nqc\" (UniqueName: \"kubernetes.io/projected/48121725-2569-4e97-b2e1-ca826ca44fe0-kube-api-access-x4nqc\") pod \"48121725-2569-4e97-b2e1-ca826ca44fe0\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") "
Sep 16 20:00:35.668430 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.668121    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48121725-2569-4e97-b2e1-ca826ca44fe0-tls-certs\") pod \"48121725-2569-4e97-b2e1-ca826ca44fe0\" (UID: \"48121725-2569-4e97-b2e1-ca826ca44fe0\") "
Sep 16 20:00:35.668430 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.668245    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-model-cache" (OuterVolumeSpecName: "model-cache") pod "48121725-2569-4e97-b2e1-ca826ca44fe0" (UID: "48121725-2569-4e97-b2e1-ca826ca44fe0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:35.668430 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.668339    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48121725-2569-4e97-b2e1-ca826ca44fe0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "48121725-2569-4e97-b2e1-ca826ca44fe0" (UID: "48121725-2569-4e97-b2e1-ca826ca44fe0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:00:35.668627 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.668502    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48121725-2569-4e97-b2e1-ca826ca44fe0-cabundle-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.668627 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.668522    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.668932 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.668887    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-home" (OuterVolumeSpecName: "home") pod "48121725-2569-4e97-b2e1-ca826ca44fe0" (UID: "48121725-2569-4e97-b2e1-ca826ca44fe0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:35.670713 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.670689    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-dshm" (OuterVolumeSpecName: "dshm") pod "48121725-2569-4e97-b2e1-ca826ca44fe0" (UID: "48121725-2569-4e97-b2e1-ca826ca44fe0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:35.670967 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.670949    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48121725-2569-4e97-b2e1-ca826ca44fe0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48121725-2569-4e97-b2e1-ca826ca44fe0" (UID: "48121725-2569-4e97-b2e1-ca826ca44fe0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:00:35.671040 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.670997    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48121725-2569-4e97-b2e1-ca826ca44fe0-kube-api-access-x4nqc" (OuterVolumeSpecName: "kube-api-access-x4nqc") pod "48121725-2569-4e97-b2e1-ca826ca44fe0" (UID: "48121725-2569-4e97-b2e1-ca826ca44fe0"). InnerVolumeSpecName "kube-api-access-x4nqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:00:35.681273 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.681244    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48121725-2569-4e97-b2e1-ca826ca44fe0" (UID: "48121725-2569-4e97-b2e1-ca826ca44fe0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:35.692483 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.692454    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48121725-2569-4e97-b2e1-ca826ca44fe0" (UID: "48121725-2569-4e97-b2e1-ca826ca44fe0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:35.769863 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.769822    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4nqc\" (UniqueName: \"kubernetes.io/projected/48121725-2569-4e97-b2e1-ca826ca44fe0-kube-api-access-x4nqc\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.769863 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.769858    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48121725-2569-4e97-b2e1-ca826ca44fe0-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.769863 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.769869    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.770115 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.769879    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.770115 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.769887    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.770115 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:35.769895    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48121725-2569-4e97-b2e1-ca826ca44fe0-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:00:36.317170 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.317133    2582 generic.go:356] "Generic (PLEG): container finished" podID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerID="ca00cf6cc31a6387f69d4ede6fcdab8d4098de96f59745cb91449b99620f9350" exitCode=0
Sep 16 20:00:36.317343 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.317195    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"
Sep 16 20:00:36.317343 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.317217    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" event={"ID":"48121725-2569-4e97-b2e1-ca826ca44fe0","Type":"ContainerDied","Data":"ca00cf6cc31a6387f69d4ede6fcdab8d4098de96f59745cb91449b99620f9350"}
Sep 16 20:00:36.317343 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.317273    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289" event={"ID":"48121725-2569-4e97-b2e1-ca826ca44fe0","Type":"ContainerDied","Data":"3609f3761fd9a5ef5721037c2c224836261f13dd4404a894cf2d80658da920c8"}
Sep 16 20:00:36.317343 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.317298    2582 scope.go:117] "RemoveContainer" containerID="d5dc5474adf18918074878f6f020454360f4fe31d43b66bbbd695f273a54a052"
Sep 16 20:00:36.327497 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.327478    2582 scope.go:117] "RemoveContainer" containerID="09764b2d520bb503997f606a629bdb532985e34639f1f9c0328ec32247bcb29f"
Sep 16 20:00:36.336077 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.336059    2582 scope.go:117] "RemoveContainer" containerID="ca00cf6cc31a6387f69d4ede6fcdab8d4098de96f59745cb91449b99620f9350"
Sep 16 20:00:36.343782 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.343753    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"]
Sep 16 20:00:36.344169 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.344147    2582 scope.go:117] "RemoveContainer" containerID="d5dc5474adf18918074878f6f020454360f4fe31d43b66bbbd695f273a54a052"
Sep 16 20:00:36.344448 ip-10-0-142-173 kubenswrapper[2582]: E0916 20:00:36.344428    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5dc5474adf18918074878f6f020454360f4fe31d43b66bbbd695f273a54a052\": container with ID starting with d5dc5474adf18918074878f6f020454360f4fe31d43b66bbbd695f273a54a052 not found: ID does not exist" containerID="d5dc5474adf18918074878f6f020454360f4fe31d43b66bbbd695f273a54a052"
Sep 16 20:00:36.344500 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.344459    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5dc5474adf18918074878f6f020454360f4fe31d43b66bbbd695f273a54a052"} err="failed to get container status \"d5dc5474adf18918074878f6f020454360f4fe31d43b66bbbd695f273a54a052\": rpc error: code = NotFound desc = could not find container \"d5dc5474adf18918074878f6f020454360f4fe31d43b66bbbd695f273a54a052\": container with ID starting with d5dc5474adf18918074878f6f020454360f4fe31d43b66bbbd695f273a54a052 not found: ID does not exist"
Sep 16 20:00:36.344500 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.344483    2582 scope.go:117] "RemoveContainer" containerID="09764b2d520bb503997f606a629bdb532985e34639f1f9c0328ec32247bcb29f"
Sep 16 20:00:36.344762 ip-10-0-142-173 kubenswrapper[2582]: E0916 20:00:36.344745    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09764b2d520bb503997f606a629bdb532985e34639f1f9c0328ec32247bcb29f\": container with ID starting with 09764b2d520bb503997f606a629bdb532985e34639f1f9c0328ec32247bcb29f not found: ID does not exist" containerID="09764b2d520bb503997f606a629bdb532985e34639f1f9c0328ec32247bcb29f"
Sep 16 20:00:36.344820 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.344769    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09764b2d520bb503997f606a629bdb532985e34639f1f9c0328ec32247bcb29f"} err="failed to get container status \"09764b2d520bb503997f606a629bdb532985e34639f1f9c0328ec32247bcb29f\": rpc error: code = NotFound desc = could not find container \"09764b2d520bb503997f606a629bdb532985e34639f1f9c0328ec32247bcb29f\": container with ID starting with 09764b2d520bb503997f606a629bdb532985e34639f1f9c0328ec32247bcb29f not found: ID does not exist"
Sep 16 20:00:36.344820 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.344786    2582 scope.go:117] "RemoveContainer" containerID="ca00cf6cc31a6387f69d4ede6fcdab8d4098de96f59745cb91449b99620f9350"
Sep 16 20:00:36.345036 ip-10-0-142-173 kubenswrapper[2582]: E0916 20:00:36.345022    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca00cf6cc31a6387f69d4ede6fcdab8d4098de96f59745cb91449b99620f9350\": container with ID starting with ca00cf6cc31a6387f69d4ede6fcdab8d4098de96f59745cb91449b99620f9350 not found: ID does not exist" containerID="ca00cf6cc31a6387f69d4ede6fcdab8d4098de96f59745cb91449b99620f9350"
Sep 16 20:00:36.345074 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.345040    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca00cf6cc31a6387f69d4ede6fcdab8d4098de96f59745cb91449b99620f9350"} err="failed to get container status \"ca00cf6cc31a6387f69d4ede6fcdab8d4098de96f59745cb91449b99620f9350\": rpc error: code = NotFound desc = could not find container \"ca00cf6cc31a6387f69d4ede6fcdab8d4098de96f59745cb91449b99620f9350\": container with ID starting with ca00cf6cc31a6387f69d4ede6fcdab8d4098de96f59745cb91449b99620f9350 not found: ID does not exist"
Sep 16 20:00:36.353838 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.353809    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6fc447d467-8n289"]
Sep 16 20:00:36.704283 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:36.704181    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" path="/var/lib/kubelet/pods/48121725-2569-4e97-b2e1-ca826ca44fe0/volumes"
Sep 16 20:00:38.587561 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:38.587518    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz" podUID="11cb76bb-0f84-48fc-ac45-177e2e9db0b2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:45.818036 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.817995    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"]
Sep 16 20:00:45.818455 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.818386    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main"
Sep 16 20:00:45.818455 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.818398    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main"
Sep 16 20:00:45.818455 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.818413    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="llm-d-routing-sidecar"
Sep 16 20:00:45.818455 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.818419    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="llm-d-routing-sidecar"
Sep 16 20:00:45.818455 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.818431    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="storage-initializer"
Sep 16 20:00:45.818455 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.818437    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="storage-initializer"
Sep 16 20:00:45.818697 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.818506    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="main"
Sep 16 20:00:45.818697 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.818515    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="48121725-2569-4e97-b2e1-ca826ca44fe0" containerName="llm-d-routing-sidecar"
Sep 16 20:00:45.823691 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.823673    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:45.826951 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.826919    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 20:00:45.826951 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.826947    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:00:45.827163 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.827005    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 20:00:45.827163 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.827059    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 20:00:45.827163 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.827004    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-42p9m\""
Sep 16 20:00:45.827163 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.827141    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 20:00:45.833285 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.833263    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"]
Sep 16 20:00:45.957158 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.957117    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5xqg\" (UniqueName: \"kubernetes.io/projected/9cc070f6-2497-4511-8ac4-b84e528b8114-kube-api-access-r5xqg\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:45.957351 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.957223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-model-cache\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:45.957351 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.957240    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-tmp-dir\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:45.957351 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.957280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-home\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:45.957351 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.957306    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cc070f6-2497-4511-8ac4-b84e528b8114-tls-certs\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:45.957351 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.957321    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cc070f6-2497-4511-8ac4-b84e528b8114-cabundle-cert\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:45.957526 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.957357    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-dshm\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:45.957526 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:45.957374    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-kserve-provision-location\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.058466 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.058420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-dshm\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.058697 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.058481    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-kserve-provision-location\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.058697 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.058519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xqg\" (UniqueName: \"kubernetes.io/projected/9cc070f6-2497-4511-8ac4-b84e528b8114-kube-api-access-r5xqg\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.058697 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.058665    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-model-cache\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.058697 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.058695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-tmp-dir\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.058931 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.058744    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-home\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.058931 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.058770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cc070f6-2497-4511-8ac4-b84e528b8114-tls-certs\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.058931 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.058793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cc070f6-2497-4511-8ac4-b84e528b8114-cabundle-cert\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.059093 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.058976    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-kserve-provision-location\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.059093 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.059034    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-tmp-dir\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.059203 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.059098    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-home\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.059265 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.059225    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-model-cache\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.059395 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.059377    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cc070f6-2497-4511-8ac4-b84e528b8114-cabundle-cert\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.061048 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.061027    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-dshm\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.061369 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.061352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cc070f6-2497-4511-8ac4-b84e528b8114-tls-certs\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.069452 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.069379    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xqg\" (UniqueName: \"kubernetes.io/projected/9cc070f6-2497-4511-8ac4-b84e528b8114-kube-api-access-r5xqg\") pod \"prestop-hook-test-kserve-fff447946-x2w8h\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.135892 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.135841    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:46.419498 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:46.419471    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"]
Sep 16 20:00:46.421847 ip-10-0-142-173 kubenswrapper[2582]: W0916 20:00:46.421808    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc070f6_2497_4511_8ac4_b84e528b8114.slice/crio-8f44c9bf9f37694bac8e89ef34afc655dfaa10ebd4a0185ab19a484bf0215319 WatchSource:0}: Error finding container 8f44c9bf9f37694bac8e89ef34afc655dfaa10ebd4a0185ab19a484bf0215319: Status 404 returned error can't find the container with id 8f44c9bf9f37694bac8e89ef34afc655dfaa10ebd4a0185ab19a484bf0215319
Sep 16 20:00:47.370553 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:47.370505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h" event={"ID":"9cc070f6-2497-4511-8ac4-b84e528b8114","Type":"ContainerStarted","Data":"a1b37185cb301b0eea80c6da18568976798f3dbde83f8da60d1bdb3d0e01bd2d"}
Sep 16 20:00:47.370978 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:47.370639    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h" event={"ID":"9cc070f6-2497-4511-8ac4-b84e528b8114","Type":"ContainerStarted","Data":"8f44c9bf9f37694bac8e89ef34afc655dfaa10ebd4a0185ab19a484bf0215319"}
Sep 16 20:00:48.587770 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:48.587724    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz" podUID="11cb76bb-0f84-48fc-ac45-177e2e9db0b2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:48.588228 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:48.587888    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"
Sep 16 20:00:49.382801 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:49.382769    2582 generic.go:356] "Generic (PLEG): container finished" podID="9cc070f6-2497-4511-8ac4-b84e528b8114" containerID="a1b37185cb301b0eea80c6da18568976798f3dbde83f8da60d1bdb3d0e01bd2d" exitCode=0
Sep 16 20:00:49.383026 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:49.382849    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h" event={"ID":"9cc070f6-2497-4511-8ac4-b84e528b8114","Type":"ContainerDied","Data":"a1b37185cb301b0eea80c6da18568976798f3dbde83f8da60d1bdb3d0e01bd2d"}
Sep 16 20:00:50.391258 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:50.391218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h" event={"ID":"9cc070f6-2497-4511-8ac4-b84e528b8114","Type":"ContainerStarted","Data":"69f407a96ef3ce19ba6b0b4a9449506df99c7cef2d54819213ab69bd91c438d8"}
Sep 16 20:00:50.415009 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:50.414952    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h" podStartSLOduration=3.118227712 podStartE2EDuration="5.414929366s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.296701654s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:00:50.412822009 +0000 UTC m=+4508.243430554" watchObservedRunningTime="2026-09-16 20:00:50.414929366 +0000 UTC m=+4508.245537913"
Sep 16 20:00:51.136606 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:51.136522    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:56.136023 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:56.135973    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:00:56.137637 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:56.137604    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h" podUID="9cc070f6-2497-4511-8ac4-b84e528b8114" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": dial tcp 10.134.0.91:8000: connect: connection refused"
Sep 16 20:00:58.588386 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:00:58.588349    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz" podUID="11cb76bb-0f84-48fc-ac45-177e2e9db0b2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:04.154831 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:04.154786    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"]
Sep 16 20:01:04.161489 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:04.161438    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:04.239649 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:04.239611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:04.928942 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:04.928915    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"
Sep 16 20:01:05.033507 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.033472    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sflg\" (UniqueName: \"kubernetes.io/projected/11cb76bb-0f84-48fc-ac45-177e2e9db0b2-kube-api-access-2sflg\") pod \"11cb76bb-0f84-48fc-ac45-177e2e9db0b2\" (UID: \"11cb76bb-0f84-48fc-ac45-177e2e9db0b2\") "
Sep 16 20:01:05.033713 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.033663    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cb76bb-0f84-48fc-ac45-177e2e9db0b2-tls-certs\") pod \"11cb76bb-0f84-48fc-ac45-177e2e9db0b2\" (UID: \"11cb76bb-0f84-48fc-ac45-177e2e9db0b2\") "
Sep 16 20:01:05.036029 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.035988    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11cb76bb-0f84-48fc-ac45-177e2e9db0b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11cb76bb-0f84-48fc-ac45-177e2e9db0b2" (UID: "11cb76bb-0f84-48fc-ac45-177e2e9db0b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:01:05.036158 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.035997    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11cb76bb-0f84-48fc-ac45-177e2e9db0b2-kube-api-access-2sflg" (OuterVolumeSpecName: "kube-api-access-2sflg") pod "11cb76bb-0f84-48fc-ac45-177e2e9db0b2" (UID: "11cb76bb-0f84-48fc-ac45-177e2e9db0b2"). InnerVolumeSpecName "kube-api-access-2sflg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:05.135434 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.135386    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cb76bb-0f84-48fc-ac45-177e2e9db0b2-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:01:05.135434 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.135430    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sflg\" (UniqueName: \"kubernetes.io/projected/11cb76bb-0f84-48fc-ac45-177e2e9db0b2-kube-api-access-2sflg\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:01:05.468977 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.468883    2582 generic.go:356] "Generic (PLEG): container finished" podID="11cb76bb-0f84-48fc-ac45-177e2e9db0b2" containerID="409b78144abcb9851259096ea50cfdd9e6202210f5b0023bed9c67961a0a0d32" exitCode=0
Sep 16 20:01:05.468977 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.468944    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"
Sep 16 20:01:05.468977 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.468967    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz" event={"ID":"11cb76bb-0f84-48fc-ac45-177e2e9db0b2","Type":"ContainerDied","Data":"409b78144abcb9851259096ea50cfdd9e6202210f5b0023bed9c67961a0a0d32"}
Sep 16 20:01:05.469554 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.469030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz" event={"ID":"11cb76bb-0f84-48fc-ac45-177e2e9db0b2","Type":"ContainerDied","Data":"f9e54d665b7dbb533b9019f44a5e495c3110e2fc7d694e772d9c359f40fe9089"}
Sep 16 20:01:05.469554 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.469061    2582 scope.go:117] "RemoveContainer" containerID="409b78144abcb9851259096ea50cfdd9e6202210f5b0023bed9c67961a0a0d32"
Sep 16 20:01:05.480351 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.480329    2582 scope.go:117] "RemoveContainer" containerID="409b78144abcb9851259096ea50cfdd9e6202210f5b0023bed9c67961a0a0d32"
Sep 16 20:01:05.480666 ip-10-0-142-173 kubenswrapper[2582]: E0916 20:01:05.480647    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409b78144abcb9851259096ea50cfdd9e6202210f5b0023bed9c67961a0a0d32\": container with ID starting with 409b78144abcb9851259096ea50cfdd9e6202210f5b0023bed9c67961a0a0d32 not found: ID does not exist" containerID="409b78144abcb9851259096ea50cfdd9e6202210f5b0023bed9c67961a0a0d32"
Sep 16 20:01:05.480728 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.480679    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409b78144abcb9851259096ea50cfdd9e6202210f5b0023bed9c67961a0a0d32"} err="failed to get container status \"409b78144abcb9851259096ea50cfdd9e6202210f5b0023bed9c67961a0a0d32\": rpc error: code = NotFound desc = could not find container \"409b78144abcb9851259096ea50cfdd9e6202210f5b0023bed9c67961a0a0d32\": container with ID starting with 409b78144abcb9851259096ea50cfdd9e6202210f5b0023bed9c67961a0a0d32 not found: ID does not exist"
Sep 16 20:01:05.494777 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.494749    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"]
Sep 16 20:01:05.497452 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:05.497422    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454mqhz"]
Sep 16 20:01:06.705588 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:06.705540    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11cb76bb-0f84-48fc-ac45-177e2e9db0b2" path="/var/lib/kubelet/pods/11cb76bb-0f84-48fc-ac45-177e2e9db0b2/volumes"
Sep 16 20:01:09.796449 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:09.796412    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:09.903119 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:09.903084    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:14.742414 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:14.742374    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"]
Sep 16 20:01:14.744082 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:14.742614    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m" podUID="4fa1270e-8c20-41ba-ba08-2c939402ef54" containerName="istio-proxy" containerID="cri-o://e85d312a77462725d18361c30a350b7fc301e6f5af8b996dc7399ddc77c25030" gracePeriod=30
Sep 16 20:01:14.837772 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:14.837732    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:16.814691 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:16.814650    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m" podUID="4fa1270e-8c20-41ba-ba08-2c939402ef54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 20:01:19.179027 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:19.178982    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h" podUID="9cc070f6-2497-4511-8ac4-b84e528b8114" containerName="main" containerID="cri-o://69f407a96ef3ce19ba6b0b4a9449506df99c7cef2d54819213ab69bd91c438d8" gracePeriod=45
Sep 16 20:01:19.994770 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:19.994742    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 20:01:20.086359 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.086322    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-podinfo\") pod \"4fa1270e-8c20-41ba-ba08-2c939402ef54\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") "
Sep 16 20:01:20.086543 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.086380    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa1270e-8c20-41ba-ba08-2c939402ef54-istiod-ca-cert\") pod \"4fa1270e-8c20-41ba-ba08-2c939402ef54\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") "
Sep 16 20:01:20.086543 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.086437    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-workload-socket\") pod \"4fa1270e-8c20-41ba-ba08-2c939402ef54\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") "
Sep 16 20:01:20.086543 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.086464    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-token\") pod \"4fa1270e-8c20-41ba-ba08-2c939402ef54\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") "
Sep 16 20:01:20.086543 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.086487    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-envoy\") pod \"4fa1270e-8c20-41ba-ba08-2c939402ef54\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") "
Sep 16 20:01:20.086543 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.086529    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-workload-certs\") pod \"4fa1270e-8c20-41ba-ba08-2c939402ef54\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") "
Sep 16 20:01:20.086822 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.086588    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-credential-socket\") pod \"4fa1270e-8c20-41ba-ba08-2c939402ef54\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") "
Sep 16 20:01:20.086822 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.086618    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-data\") pod \"4fa1270e-8c20-41ba-ba08-2c939402ef54\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") "
Sep 16 20:01:20.086822 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.086641    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd7f4\" (UniqueName: \"kubernetes.io/projected/4fa1270e-8c20-41ba-ba08-2c939402ef54-kube-api-access-vd7f4\") pod \"4fa1270e-8c20-41ba-ba08-2c939402ef54\" (UID: \"4fa1270e-8c20-41ba-ba08-2c939402ef54\") "
Sep 16 20:01:20.086997 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.086864    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4fa1270e-8c20-41ba-ba08-2c939402ef54" (UID: "4fa1270e-8c20-41ba-ba08-2c939402ef54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:20.087367 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.086880    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa1270e-8c20-41ba-ba08-2c939402ef54-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4fa1270e-8c20-41ba-ba08-2c939402ef54" (UID: "4fa1270e-8c20-41ba-ba08-2c939402ef54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:01:20.087367 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.086949    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4fa1270e-8c20-41ba-ba08-2c939402ef54" (UID: "4fa1270e-8c20-41ba-ba08-2c939402ef54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:20.087367 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.087150    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-data" (OuterVolumeSpecName: "istio-data") pod "4fa1270e-8c20-41ba-ba08-2c939402ef54" (UID: "4fa1270e-8c20-41ba-ba08-2c939402ef54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:20.087367 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.087028    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-workload-socket\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.087367 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.087288    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4fa1270e-8c20-41ba-ba08-2c939402ef54" (UID: "4fa1270e-8c20-41ba-ba08-2c939402ef54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:20.089007 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.088978    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4fa1270e-8c20-41ba-ba08-2c939402ef54" (UID: "4fa1270e-8c20-41ba-ba08-2c939402ef54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 20:01:20.089007 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.088991    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-token" (OuterVolumeSpecName: "istio-token") pod "4fa1270e-8c20-41ba-ba08-2c939402ef54" (UID: "4fa1270e-8c20-41ba-ba08-2c939402ef54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:20.089236 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.089220    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa1270e-8c20-41ba-ba08-2c939402ef54-kube-api-access-vd7f4" (OuterVolumeSpecName: "kube-api-access-vd7f4") pod "4fa1270e-8c20-41ba-ba08-2c939402ef54" (UID: "4fa1270e-8c20-41ba-ba08-2c939402ef54"). InnerVolumeSpecName "kube-api-access-vd7f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:20.089614 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.089567    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4fa1270e-8c20-41ba-ba08-2c939402ef54" (UID: "4fa1270e-8c20-41ba-ba08-2c939402ef54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:20.188352 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.188310    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-data\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.188352 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.188347    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd7f4\" (UniqueName: \"kubernetes.io/projected/4fa1270e-8c20-41ba-ba08-2c939402ef54-kube-api-access-vd7f4\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.188352 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.188358    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-podinfo\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.188844 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.188367    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa1270e-8c20-41ba-ba08-2c939402ef54-istiod-ca-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.188844 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.188375    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-token\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.188844 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.188382    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-istio-envoy\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.188844 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.188390    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-workload-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.188844 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.188398    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa1270e-8c20-41ba-ba08-2c939402ef54-credential-socket\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:01:20.450399 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.450310    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:20.545182 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.545143    2582 generic.go:356] "Generic (PLEG): container finished" podID="4fa1270e-8c20-41ba-ba08-2c939402ef54" containerID="e85d312a77462725d18361c30a350b7fc301e6f5af8b996dc7399ddc77c25030" exitCode=0
Sep 16 20:01:20.545376 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.545209    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"
Sep 16 20:01:20.545376 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.545218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m" event={"ID":"4fa1270e-8c20-41ba-ba08-2c939402ef54","Type":"ContainerDied","Data":"e85d312a77462725d18361c30a350b7fc301e6f5af8b996dc7399ddc77c25030"}
Sep 16 20:01:20.545376 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.545303    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m" event={"ID":"4fa1270e-8c20-41ba-ba08-2c939402ef54","Type":"ContainerDied","Data":"a032bc892f66f49630179f5cf7a7c909cc0b314bdfac5a8bbd0347e987793117"}
Sep 16 20:01:20.545376 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.545326    2582 scope.go:117] "RemoveContainer" containerID="e85d312a77462725d18361c30a350b7fc301e6f5af8b996dc7399ddc77c25030"
Sep 16 20:01:20.554525 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.554500    2582 scope.go:117] "RemoveContainer" containerID="e85d312a77462725d18361c30a350b7fc301e6f5af8b996dc7399ddc77c25030"
Sep 16 20:01:20.554832 ip-10-0-142-173 kubenswrapper[2582]: E0916 20:01:20.554813    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85d312a77462725d18361c30a350b7fc301e6f5af8b996dc7399ddc77c25030\": container with ID starting with e85d312a77462725d18361c30a350b7fc301e6f5af8b996dc7399ddc77c25030 not found: ID does not exist" containerID="e85d312a77462725d18361c30a350b7fc301e6f5af8b996dc7399ddc77c25030"
Sep 16 20:01:20.554894 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.554840    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85d312a77462725d18361c30a350b7fc301e6f5af8b996dc7399ddc77c25030"} err="failed to get container status \"e85d312a77462725d18361c30a350b7fc301e6f5af8b996dc7399ddc77c25030\": rpc error: code = NotFound desc = could not find container \"e85d312a77462725d18361c30a350b7fc301e6f5af8b996dc7399ddc77c25030\": container with ID starting with e85d312a77462725d18361c30a350b7fc301e6f5af8b996dc7399ddc77c25030 not found: ID does not exist"
Sep 16 20:01:20.572481 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.572452    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"]
Sep 16 20:01:20.577302 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.577259    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5858dff66d-fpz2m"]
Sep 16 20:01:20.703918 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:20.703829    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fa1270e-8c20-41ba-ba08-2c939402ef54" path="/var/lib/kubelet/pods/4fa1270e-8c20-41ba-ba08-2c939402ef54/volumes"
Sep 16 20:01:34.137395 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:34.137350    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:37.344772 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:37.344729    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:37.445356 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:37.445309    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:43.571312 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:01:43.571265    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:03.750072 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:03.750020    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:04.316626 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.316592    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:04.391025 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.390996    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:04.773159 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.773136    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-fff447946-x2w8h_9cc070f6-2497-4511-8ac4-b84e528b8114/main/0.log"
Sep 16 20:02:04.773630 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.773605    2582 generic.go:356] "Generic (PLEG): container finished" podID="9cc070f6-2497-4511-8ac4-b84e528b8114" containerID="69f407a96ef3ce19ba6b0b4a9449506df99c7cef2d54819213ab69bd91c438d8" exitCode=137
Sep 16 20:02:04.773690 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.773648    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h" event={"ID":"9cc070f6-2497-4511-8ac4-b84e528b8114","Type":"ContainerDied","Data":"69f407a96ef3ce19ba6b0b4a9449506df99c7cef2d54819213ab69bd91c438d8"}
Sep 16 20:02:04.817902 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.817885    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-fff447946-x2w8h_9cc070f6-2497-4511-8ac4-b84e528b8114/main/0.log"
Sep 16 20:02:04.820409 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.820391    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:02:04.901947 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.901924    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-home\") pod \"9cc070f6-2497-4511-8ac4-b84e528b8114\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") "
Sep 16 20:02:04.902043 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.901959    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-model-cache\") pod \"9cc070f6-2497-4511-8ac4-b84e528b8114\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") "
Sep 16 20:02:04.902043 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.901990    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-tmp-dir\") pod \"9cc070f6-2497-4511-8ac4-b84e528b8114\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") "
Sep 16 20:02:04.902133 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.902083    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5xqg\" (UniqueName: \"kubernetes.io/projected/9cc070f6-2497-4511-8ac4-b84e528b8114-kube-api-access-r5xqg\") pod \"9cc070f6-2497-4511-8ac4-b84e528b8114\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") "
Sep 16 20:02:04.902133 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.902117    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cc070f6-2497-4511-8ac4-b84e528b8114-cabundle-cert\") pod \"9cc070f6-2497-4511-8ac4-b84e528b8114\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") "
Sep 16 20:02:04.902226 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.902142    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-kserve-provision-location\") pod \"9cc070f6-2497-4511-8ac4-b84e528b8114\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") "
Sep 16 20:02:04.902226 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.902164    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-dshm\") pod \"9cc070f6-2497-4511-8ac4-b84e528b8114\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") "
Sep 16 20:02:04.902226 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.902186    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-model-cache" (OuterVolumeSpecName: "model-cache") pod "9cc070f6-2497-4511-8ac4-b84e528b8114" (UID: "9cc070f6-2497-4511-8ac4-b84e528b8114"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:04.902226 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.902199    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cc070f6-2497-4511-8ac4-b84e528b8114-tls-certs\") pod \"9cc070f6-2497-4511-8ac4-b84e528b8114\" (UID: \"9cc070f6-2497-4511-8ac4-b84e528b8114\") "
Sep 16 20:02:04.902448 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.902426    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:02:04.902508 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.902488    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-home" (OuterVolumeSpecName: "home") pod "9cc070f6-2497-4511-8ac4-b84e528b8114" (UID: "9cc070f6-2497-4511-8ac4-b84e528b8114"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:04.902565 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.902502    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9cc070f6-2497-4511-8ac4-b84e528b8114" (UID: "9cc070f6-2497-4511-8ac4-b84e528b8114"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:04.902565 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.902534    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc070f6-2497-4511-8ac4-b84e528b8114-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9cc070f6-2497-4511-8ac4-b84e528b8114" (UID: "9cc070f6-2497-4511-8ac4-b84e528b8114"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:02:04.904531 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.904501    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc070f6-2497-4511-8ac4-b84e528b8114-kube-api-access-r5xqg" (OuterVolumeSpecName: "kube-api-access-r5xqg") pod "9cc070f6-2497-4511-8ac4-b84e528b8114" (UID: "9cc070f6-2497-4511-8ac4-b84e528b8114"). InnerVolumeSpecName "kube-api-access-r5xqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:02:04.904531 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.904514    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc070f6-2497-4511-8ac4-b84e528b8114-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9cc070f6-2497-4511-8ac4-b84e528b8114" (UID: "9cc070f6-2497-4511-8ac4-b84e528b8114"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:02:04.904813 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.904556    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-dshm" (OuterVolumeSpecName: "dshm") pod "9cc070f6-2497-4511-8ac4-b84e528b8114" (UID: "9cc070f6-2497-4511-8ac4-b84e528b8114"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:04.924797 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:04.924776    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9cc070f6-2497-4511-8ac4-b84e528b8114" (UID: "9cc070f6-2497-4511-8ac4-b84e528b8114"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:05.003474 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:05.003414    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5xqg\" (UniqueName: \"kubernetes.io/projected/9cc070f6-2497-4511-8ac4-b84e528b8114-kube-api-access-r5xqg\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:02:05.003474 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:05.003437    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cc070f6-2497-4511-8ac4-b84e528b8114-cabundle-cert\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:02:05.003474 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:05.003446    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-kserve-provision-location\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:02:05.003474 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:05.003455    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:02:05.003474 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:05.003464    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cc070f6-2497-4511-8ac4-b84e528b8114-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:02:05.003474 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:05.003473    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:02:05.003743 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:05.003480    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cc070f6-2497-4511-8ac4-b84e528b8114-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:02:05.779148 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:05.779122    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-fff447946-x2w8h_9cc070f6-2497-4511-8ac4-b84e528b8114/main/0.log"
Sep 16 20:02:05.779564 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:05.779535    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"
Sep 16 20:02:05.779564 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:05.779543    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h" event={"ID":"9cc070f6-2497-4511-8ac4-b84e528b8114","Type":"ContainerDied","Data":"8f44c9bf9f37694bac8e89ef34afc655dfaa10ebd4a0185ab19a484bf0215319"}
Sep 16 20:02:05.779677 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:05.779614    2582 scope.go:117] "RemoveContainer" containerID="69f407a96ef3ce19ba6b0b4a9449506df99c7cef2d54819213ab69bd91c438d8"
Sep 16 20:02:05.789152 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:05.789131    2582 scope.go:117] "RemoveContainer" containerID="a1b37185cb301b0eea80c6da18568976798f3dbde83f8da60d1bdb3d0e01bd2d"
Sep 16 20:02:05.804656 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:05.804631    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"]
Sep 16 20:02:05.808637 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:05.808616    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-x2w8h"]
Sep 16 20:02:06.703634 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:06.703601    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cc070f6-2497-4511-8ac4-b84e528b8114" path="/var/lib/kubelet/pods/9cc070f6-2497-4511-8ac4-b84e528b8114/volumes"
Sep 16 20:02:09.340635 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:09.340604    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:09.455606 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:09.455563    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:14.013784 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.013748    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"]
Sep 16 20:02:14.014172 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.014055    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11cb76bb-0f84-48fc-ac45-177e2e9db0b2" containerName="main"
Sep 16 20:02:14.014172 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.014065    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="11cb76bb-0f84-48fc-ac45-177e2e9db0b2" containerName="main"
Sep 16 20:02:14.014172 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.014076    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fa1270e-8c20-41ba-ba08-2c939402ef54" containerName="istio-proxy"
Sep 16 20:02:14.014172 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.014081    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fa1270e-8c20-41ba-ba08-2c939402ef54" containerName="istio-proxy"
Sep 16 20:02:14.014172 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.014087    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cc070f6-2497-4511-8ac4-b84e528b8114" containerName="main"
Sep 16 20:02:14.014172 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.014093    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cc070f6-2497-4511-8ac4-b84e528b8114" containerName="main"
Sep 16 20:02:14.014172 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.014110    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cc070f6-2497-4511-8ac4-b84e528b8114" containerName="storage-initializer"
Sep 16 20:02:14.014172 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.014115    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cc070f6-2497-4511-8ac4-b84e528b8114" containerName="storage-initializer"
Sep 16 20:02:14.014172 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.014162    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="11cb76bb-0f84-48fc-ac45-177e2e9db0b2" containerName="main"
Sep 16 20:02:14.014172 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.014173    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fa1270e-8c20-41ba-ba08-2c939402ef54" containerName="istio-proxy"
Sep 16 20:02:14.014172 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.014180    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cc070f6-2497-4511-8ac4-b84e528b8114" containerName="main"
Sep 16 20:02:14.018456 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.018439    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.022625 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.022608    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 20:02:14.022739 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.022647    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 20:02:14.022739 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.022652    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 20:02:14.022739 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.022647    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-tmh5s\""
Sep 16 20:02:14.028087 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.028065    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"]
Sep 16 20:02:14.071428 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.071401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.071596 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.071441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2csl\" (UniqueName: \"kubernetes.io/projected/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-kube-api-access-p2csl\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.071596 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.071490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.071596 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.071532    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.071596 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.071560    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.071850 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.071605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.104235 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.104210    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:14.172308 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.172283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.172396 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.172318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.172396 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.172340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.172492 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.172473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.172549 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.172533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.172621 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.172595    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2csl\" (UniqueName: \"kubernetes.io/projected/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-kube-api-access-p2csl\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.172699 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.172675    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.172818 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.172722    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.172893 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.172852    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.174760 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.174742    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.175013 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.174996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.181375 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.181354    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2csl\" (UniqueName: \"kubernetes.io/projected/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-kube-api-access-p2csl\") pod \"rolling-upgrade-test-kserve-74b79d7f77-lvw7z\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.270358 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.270330    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"]
Sep 16 20:02:14.273673 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.273651    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"
Sep 16 20:02:14.278465 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.278446    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-5spxx\""
Sep 16 20:02:14.287924 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.287901    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"]
Sep 16 20:02:14.315234 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.315201    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:14.329161 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.329140    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:14.374765 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.374732    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnd2s\" (UniqueName: \"kubernetes.io/projected/e3bdb7c2-af02-4701-94a8-e7f4cd398779-kube-api-access-rnd2s\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m\" (UID: \"e3bdb7c2-af02-4701-94a8-e7f4cd398779\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"
Sep 16 20:02:14.374898 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.374850    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3bdb7c2-af02-4701-94a8-e7f4cd398779-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m\" (UID: \"e3bdb7c2-af02-4701-94a8-e7f4cd398779\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"
Sep 16 20:02:14.476227 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.476187    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3bdb7c2-af02-4701-94a8-e7f4cd398779-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m\" (UID: \"e3bdb7c2-af02-4701-94a8-e7f4cd398779\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"
Sep 16 20:02:14.476364 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.476253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnd2s\" (UniqueName: \"kubernetes.io/projected/e3bdb7c2-af02-4701-94a8-e7f4cd398779-kube-api-access-rnd2s\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m\" (UID: \"e3bdb7c2-af02-4701-94a8-e7f4cd398779\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"
Sep 16 20:02:14.478779 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.478760    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3bdb7c2-af02-4701-94a8-e7f4cd398779-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m\" (UID: \"e3bdb7c2-af02-4701-94a8-e7f4cd398779\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"
Sep 16 20:02:14.484831 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.484806    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnd2s\" (UniqueName: \"kubernetes.io/projected/e3bdb7c2-af02-4701-94a8-e7f4cd398779-kube-api-access-rnd2s\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m\" (UID: \"e3bdb7c2-af02-4701-94a8-e7f4cd398779\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"
Sep 16 20:02:14.585037 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.584987    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"
Sep 16 20:02:14.603157 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.603134    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"]
Sep 16 20:02:14.605298 ip-10-0-142-173 kubenswrapper[2582]: W0916 20:02:14.605277    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73825c1_5c1b_4c32_bcf8_1c0c645865b2.slice/crio-7a52aecba7dce32aa454378288db4a27790e832221e5cde4c971b74d5b820f78 WatchSource:0}: Error finding container 7a52aecba7dce32aa454378288db4a27790e832221e5cde4c971b74d5b820f78: Status 404 returned error can't find the container with id 7a52aecba7dce32aa454378288db4a27790e832221e5cde4c971b74d5b820f78
Sep 16 20:02:14.817091 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.817062    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z" event={"ID":"c73825c1-5c1b-4c32-bcf8-1c0c645865b2","Type":"ContainerStarted","Data":"5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b"}
Sep 16 20:02:14.817270 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.817107    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z" event={"ID":"c73825c1-5c1b-4c32-bcf8-1c0c645865b2","Type":"ContainerStarted","Data":"7a52aecba7dce32aa454378288db4a27790e832221e5cde4c971b74d5b820f78"}
Sep 16 20:02:14.836267 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.836170    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z" podStartSLOduration=1.836154257 podStartE2EDuration="1.836154257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:02:14.833811947 +0000 UTC m=+4592.664420509" watchObservedRunningTime="2026-09-16 20:02:14.836154257 +0000 UTC m=+4592.666762802"
Sep 16 20:02:14.862947 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:14.862924    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"]
Sep 16 20:02:14.865287 ip-10-0-142-173 kubenswrapper[2582]: W0916 20:02:14.865260    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bdb7c2_af02_4701_94a8_e7f4cd398779.slice/crio-87a7f68b0749504bed7056753454d8dd5e65e54ff13ee60dea819b2390e50ef7 WatchSource:0}: Error finding container 87a7f68b0749504bed7056753454d8dd5e65e54ff13ee60dea819b2390e50ef7: Status 404 returned error can't find the container with id 87a7f68b0749504bed7056753454d8dd5e65e54ff13ee60dea819b2390e50ef7
Sep 16 20:02:15.330180 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:15.330150    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:15.822670 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:15.822635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m" event={"ID":"e3bdb7c2-af02-4701-94a8-e7f4cd398779","Type":"ContainerStarted","Data":"b637156c8207b4e54dc69f325c783ce70652f35a32a3aa3b833a300409f1e7d7"}
Sep 16 20:02:15.822812 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:15.822713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m" event={"ID":"e3bdb7c2-af02-4701-94a8-e7f4cd398779","Type":"ContainerStarted","Data":"87a7f68b0749504bed7056753454d8dd5e65e54ff13ee60dea819b2390e50ef7"}
Sep 16 20:02:15.822812 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:15.822759    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"
Sep 16 20:02:15.842829 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:15.842788    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m" podStartSLOduration=1.8427767560000001 podStartE2EDuration="1.842776756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:02:15.841295535 +0000 UTC m=+4593.671904082" watchObservedRunningTime="2026-09-16 20:02:15.842776756 +0000 UTC m=+4593.673385300"
Sep 16 20:02:16.099658 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:16.099527    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:20.609565 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:20.609491    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:21.781406 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:21.781370    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:21.889042 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:21.889015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:24.329569 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:24.329531    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:24.341779 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:24.341756    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:24.872680 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:24.872651    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:25.783422 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:25.783390    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:46.828720 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:46.828691    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"
Sep 16 20:02:59.090016 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.089978    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"]
Sep 16 20:02:59.100024 ip-10-0-142-173 kubenswrapper[2582]: E0916 20:02:59.099974    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:02:59.100024 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:02:59.100024 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 20:02:59.100024 ip-10-0-142-173 kubenswrapper[2582]:  > containerID="5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b" cmd=["/bin/sleep","15"]
Sep 16 20:02:59.100338 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.100047    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z" message="" err=<
Sep 16 20:02:59.100338 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:02:59.100338 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 20:02:59.100338 ip-10-0-142-173 kubenswrapper[2582]:  >
Sep 16 20:02:59.100338 ip-10-0-142-173 kubenswrapper[2582]: E0916 20:02:59.100064    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:02:59.100338 ip-10-0-142-173 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:02:59.100338 ip-10-0-142-173 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 16 20:02:59.100338 ip-10-0-142-173 kubenswrapper[2582]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z" podUID="c73825c1-5c1b-4c32-bcf8-1c0c645865b2" containerName="main" containerID="cri-o://5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b"
Sep 16 20:02:59.100338 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.100129    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z" podUID="c73825c1-5c1b-4c32-bcf8-1c0c645865b2" containerName="main" containerID="cri-o://5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b" gracePeriod=60
Sep 16 20:02:59.339253 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.339178    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:59.343779 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.343670    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"]
Sep 16 20:02:59.394522 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.394490    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:59.465017 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.464982    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:02:59.642461 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.642360    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-tmp-dir\") pod \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") "
Sep 16 20:02:59.642461 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.642418    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-model-cache\") pod \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") "
Sep 16 20:02:59.642461 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.642462    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-dshm\") pod \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") "
Sep 16 20:02:59.642806 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.642491    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-tls-certs\") pod \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") "
Sep 16 20:02:59.642806 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.642513    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-home\") pod \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") "
Sep 16 20:02:59.642806 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.642536    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2csl\" (UniqueName: \"kubernetes.io/projected/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-kube-api-access-p2csl\") pod \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\" (UID: \"c73825c1-5c1b-4c32-bcf8-1c0c645865b2\") "
Sep 16 20:02:59.642806 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.642725    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c73825c1-5c1b-4c32-bcf8-1c0c645865b2" (UID: "c73825c1-5c1b-4c32-bcf8-1c0c645865b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:59.642806 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.642798    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-home" (OuterVolumeSpecName: "home") pod "c73825c1-5c1b-4c32-bcf8-1c0c645865b2" (UID: "c73825c1-5c1b-4c32-bcf8-1c0c645865b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:59.642992 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.642817    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "c73825c1-5c1b-4c32-bcf8-1c0c645865b2" (UID: "c73825c1-5c1b-4c32-bcf8-1c0c645865b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:59.645062 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.645037    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-dshm" (OuterVolumeSpecName: "dshm") pod "c73825c1-5c1b-4c32-bcf8-1c0c645865b2" (UID: "c73825c1-5c1b-4c32-bcf8-1c0c645865b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:59.645062 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.645043    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c73825c1-5c1b-4c32-bcf8-1c0c645865b2" (UID: "c73825c1-5c1b-4c32-bcf8-1c0c645865b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:02:59.645211 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.645036    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-kube-api-access-p2csl" (OuterVolumeSpecName: "kube-api-access-p2csl") pod "c73825c1-5c1b-4c32-bcf8-1c0c645865b2" (UID: "c73825c1-5c1b-4c32-bcf8-1c0c645865b2"). InnerVolumeSpecName "kube-api-access-p2csl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:02:59.743953 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.743916    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2csl\" (UniqueName: \"kubernetes.io/projected/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-kube-api-access-p2csl\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:02:59.744128 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.743966    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-tmp-dir\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:02:59.744128 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.743980    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-model-cache\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:02:59.744128 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.743989    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-dshm\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:02:59.744128 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.743998    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:02:59.744128 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:02:59.744005    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c73825c1-5c1b-4c32-bcf8-1c0c645865b2-home\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:03:00.027906 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:00.027864    2582 generic.go:356] "Generic (PLEG): container finished" podID="c73825c1-5c1b-4c32-bcf8-1c0c645865b2" containerID="5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b" exitCode=0
Sep 16 20:03:00.028092 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:00.027914    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"
Sep 16 20:03:00.028092 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:00.027943    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z" event={"ID":"c73825c1-5c1b-4c32-bcf8-1c0c645865b2","Type":"ContainerDied","Data":"5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b"}
Sep 16 20:03:00.028092 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:00.028001    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z" event={"ID":"c73825c1-5c1b-4c32-bcf8-1c0c645865b2","Type":"ContainerDied","Data":"7a52aecba7dce32aa454378288db4a27790e832221e5cde4c971b74d5b820f78"}
Sep 16 20:03:00.028092 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:00.028023    2582 scope.go:117] "RemoveContainer" containerID="5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b"
Sep 16 20:03:00.037948 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:00.037925    2582 scope.go:117] "RemoveContainer" containerID="5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b"
Sep 16 20:03:00.038245 ip-10-0-142-173 kubenswrapper[2582]: E0916 20:03:00.038224    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b\": container with ID starting with 5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b not found: ID does not exist" containerID="5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b"
Sep 16 20:03:00.038294 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:00.038255    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b"} err="failed to get container status \"5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b\": rpc error: code = NotFound desc = could not find container \"5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b\": container with ID starting with 5f4644f933d16e05260587934635375a22ecc062c2f0d898b291d81bc4f3ad0b not found: ID does not exist"
Sep 16 20:03:00.051734 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:00.051701    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"]
Sep 16 20:03:00.059963 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:00.059933    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-lvw7z"]
Sep 16 20:03:00.704359 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:00.704328    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73825c1-5c1b-4c32-bcf8-1c0c645865b2" path="/var/lib/kubelet/pods/c73825c1-5c1b-4c32-bcf8-1c0c645865b2/volumes"
Sep 16 20:03:06.828226 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:06.828186    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m" podUID="e3bdb7c2-af02-4701-94a8-e7f4cd398779" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:14.365549 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:14.365484    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m" podUID="e3bdb7c2-af02-4701-94a8-e7f4cd398779" containerName="main" containerID="cri-o://b637156c8207b4e54dc69f325c783ce70652f35a32a3aa3b833a300409f1e7d7" gracePeriod=45
Sep 16 20:03:16.828457 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:16.828413    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m" podUID="e3bdb7c2-af02-4701-94a8-e7f4cd398779" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:25.017498 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:25.017457    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:26.828486 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:26.828444    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m" podUID="e3bdb7c2-af02-4701-94a8-e7f4cd398779" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:26.828925 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:26.828547    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"
Sep 16 20:03:36.829110 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:36.829024    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m" podUID="e3bdb7c2-af02-4701-94a8-e7f4cd398779" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:39.954519 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:39.954481    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:40.054095 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:40.054055    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:44.605137 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:44.605108    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"
Sep 16 20:03:44.724296 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:44.724213    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3bdb7c2-af02-4701-94a8-e7f4cd398779-tls-certs\") pod \"e3bdb7c2-af02-4701-94a8-e7f4cd398779\" (UID: \"e3bdb7c2-af02-4701-94a8-e7f4cd398779\") "
Sep 16 20:03:44.724296 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:44.724251    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnd2s\" (UniqueName: \"kubernetes.io/projected/e3bdb7c2-af02-4701-94a8-e7f4cd398779-kube-api-access-rnd2s\") pod \"e3bdb7c2-af02-4701-94a8-e7f4cd398779\" (UID: \"e3bdb7c2-af02-4701-94a8-e7f4cd398779\") "
Sep 16 20:03:44.726529 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:44.726499    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3bdb7c2-af02-4701-94a8-e7f4cd398779-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3bdb7c2-af02-4701-94a8-e7f4cd398779" (UID: "e3bdb7c2-af02-4701-94a8-e7f4cd398779"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:03:44.726673 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:44.726608    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bdb7c2-af02-4701-94a8-e7f4cd398779-kube-api-access-rnd2s" (OuterVolumeSpecName: "kube-api-access-rnd2s") pod "e3bdb7c2-af02-4701-94a8-e7f4cd398779" (UID: "e3bdb7c2-af02-4701-94a8-e7f4cd398779"). InnerVolumeSpecName "kube-api-access-rnd2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:03:44.825123 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:44.825067    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3bdb7c2-af02-4701-94a8-e7f4cd398779-tls-certs\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:03:44.825123 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:44.825113    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnd2s\" (UniqueName: \"kubernetes.io/projected/e3bdb7c2-af02-4701-94a8-e7f4cd398779-kube-api-access-rnd2s\") on node \"ip-10-0-142-173.ec2.internal\" DevicePath \"\""
Sep 16 20:03:44.863224 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:44.863183    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:45.240713 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:45.240678    2582 generic.go:356] "Generic (PLEG): container finished" podID="e3bdb7c2-af02-4701-94a8-e7f4cd398779" containerID="b637156c8207b4e54dc69f325c783ce70652f35a32a3aa3b833a300409f1e7d7" exitCode=0
Sep 16 20:03:45.240887 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:45.240722    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"
Sep 16 20:03:45.240887 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:45.240735    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m" event={"ID":"e3bdb7c2-af02-4701-94a8-e7f4cd398779","Type":"ContainerDied","Data":"b637156c8207b4e54dc69f325c783ce70652f35a32a3aa3b833a300409f1e7d7"}
Sep 16 20:03:45.240887 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:45.240787    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m" event={"ID":"e3bdb7c2-af02-4701-94a8-e7f4cd398779","Type":"ContainerDied","Data":"87a7f68b0749504bed7056753454d8dd5e65e54ff13ee60dea819b2390e50ef7"}
Sep 16 20:03:45.240887 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:45.240813    2582 scope.go:117] "RemoveContainer" containerID="b637156c8207b4e54dc69f325c783ce70652f35a32a3aa3b833a300409f1e7d7"
Sep 16 20:03:45.250237 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:45.250214    2582 scope.go:117] "RemoveContainer" containerID="b637156c8207b4e54dc69f325c783ce70652f35a32a3aa3b833a300409f1e7d7"
Sep 16 20:03:45.250554 ip-10-0-142-173 kubenswrapper[2582]: E0916 20:03:45.250535    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b637156c8207b4e54dc69f325c783ce70652f35a32a3aa3b833a300409f1e7d7\": container with ID starting with b637156c8207b4e54dc69f325c783ce70652f35a32a3aa3b833a300409f1e7d7 not found: ID does not exist" containerID="b637156c8207b4e54dc69f325c783ce70652f35a32a3aa3b833a300409f1e7d7"
Sep 16 20:03:45.250630 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:45.250563    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b637156c8207b4e54dc69f325c783ce70652f35a32a3aa3b833a300409f1e7d7"} err="failed to get container status \"b637156c8207b4e54dc69f325c783ce70652f35a32a3aa3b833a300409f1e7d7\": rpc error: code = NotFound desc = could not find container \"b637156c8207b4e54dc69f325c783ce70652f35a32a3aa3b833a300409f1e7d7\": container with ID starting with b637156c8207b4e54dc69f325c783ce70652f35a32a3aa3b833a300409f1e7d7 not found: ID does not exist"
Sep 16 20:03:45.262383 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:45.262349    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"]
Sep 16 20:03:45.266658 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:45.266626    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qs94m"]
Sep 16 20:03:45.710723 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:45.710692    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:46.704162 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:46.704125    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3bdb7c2-af02-4701-94a8-e7f4cd398779" path="/var/lib/kubelet/pods/e3bdb7c2-af02-4701-94a8-e7f4cd398779/volumes"
Sep 16 20:03:47.984163 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:47.984123    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:49.397300 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:49.397268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:50.614964 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:50.614905    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:51.445530 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:51.445487    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:56.116514 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:56.116478    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:56.216975 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:56.216945    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:57.194353 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:57.194310    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:57.292653 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:03:57.292620    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:13.327679 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:13.327636    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-nhjkq_ed9ecc5c-c5f2-46bd-96ff-c664fae2347c/authorino/0.log"
Sep 16 20:04:13.369935 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:13.369906    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-nqdvt_0e1df62f-f49b-4f8e-bdf4-0502e635115d/kuadrant-console-plugin/0.log"
Sep 16 20:04:16.944520 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:16.944482    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:17.102070 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.102011    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9zj9p/must-gather-4kqn5"]
Sep 16 20:04:17.102667 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.102632    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73825c1-5c1b-4c32-bcf8-1c0c645865b2" containerName="main"
Sep 16 20:04:17.102667 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.102661    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73825c1-5c1b-4c32-bcf8-1c0c645865b2" containerName="main"
Sep 16 20:04:17.102975 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.102693    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3bdb7c2-af02-4701-94a8-e7f4cd398779" containerName="main"
Sep 16 20:04:17.102975 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.102702    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3bdb7c2-af02-4701-94a8-e7f4cd398779" containerName="main"
Sep 16 20:04:17.102975 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.102836    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73825c1-5c1b-4c32-bcf8-1c0c645865b2" containerName="main"
Sep 16 20:04:17.102975 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.102852    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3bdb7c2-af02-4701-94a8-e7f4cd398779" containerName="main"
Sep 16 20:04:17.106388 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.106358    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zj9p/must-gather-4kqn5"
Sep 16 20:04:17.109518 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.109489    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9zj9p\"/\"openshift-service-ca.crt\""
Sep 16 20:04:17.109730 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.109561    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9zj9p\"/\"kube-root-ca.crt\""
Sep 16 20:04:17.111160 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.111135    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9zj9p\"/\"default-dockercfg-9qldv\""
Sep 16 20:04:17.116952 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.116928    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zj9p/must-gather-4kqn5"]
Sep 16 20:04:17.122643 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.122596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:17.198219 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.198104    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:17.203352 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.203320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skxd6\" (UniqueName: \"kubernetes.io/projected/b7184cf4-b9cd-4d27-80ba-5e412a3b623a-kube-api-access-skxd6\") pod \"must-gather-4kqn5\" (UID: \"b7184cf4-b9cd-4d27-80ba-5e412a3b623a\") " pod="openshift-must-gather-9zj9p/must-gather-4kqn5"
Sep 16 20:04:17.203477 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.203364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b7184cf4-b9cd-4d27-80ba-5e412a3b623a-must-gather-output\") pod \"must-gather-4kqn5\" (UID: \"b7184cf4-b9cd-4d27-80ba-5e412a3b623a\") " pod="openshift-must-gather-9zj9p/must-gather-4kqn5"
Sep 16 20:04:17.304543 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.304496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skxd6\" (UniqueName: \"kubernetes.io/projected/b7184cf4-b9cd-4d27-80ba-5e412a3b623a-kube-api-access-skxd6\") pod \"must-gather-4kqn5\" (UID: \"b7184cf4-b9cd-4d27-80ba-5e412a3b623a\") " pod="openshift-must-gather-9zj9p/must-gather-4kqn5"
Sep 16 20:04:17.304543 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.304546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b7184cf4-b9cd-4d27-80ba-5e412a3b623a-must-gather-output\") pod \"must-gather-4kqn5\" (UID: \"b7184cf4-b9cd-4d27-80ba-5e412a3b623a\") " pod="openshift-must-gather-9zj9p/must-gather-4kqn5"
Sep 16 20:04:17.304986 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.304966    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b7184cf4-b9cd-4d27-80ba-5e412a3b623a-must-gather-output\") pod \"must-gather-4kqn5\" (UID: \"b7184cf4-b9cd-4d27-80ba-5e412a3b623a\") " pod="openshift-must-gather-9zj9p/must-gather-4kqn5"
Sep 16 20:04:17.315110 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.315086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skxd6\" (UniqueName: \"kubernetes.io/projected/b7184cf4-b9cd-4d27-80ba-5e412a3b623a-kube-api-access-skxd6\") pod \"must-gather-4kqn5\" (UID: \"b7184cf4-b9cd-4d27-80ba-5e412a3b623a\") " pod="openshift-must-gather-9zj9p/must-gather-4kqn5"
Sep 16 20:04:17.417824 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.417769    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zj9p/must-gather-4kqn5"
Sep 16 20:04:17.701852 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.701777    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zj9p/must-gather-4kqn5"]
Sep 16 20:04:17.708706 ip-10-0-142-173 kubenswrapper[2582]: W0916 20:04:17.708658    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7184cf4_b9cd_4d27_80ba_5e412a3b623a.slice/crio-0ad92fe2832b1015d20c51a60cd9d0813a85d4c1e317da79f0ee260b0385671f WatchSource:0}: Error finding container 0ad92fe2832b1015d20c51a60cd9d0813a85d4c1e317da79f0ee260b0385671f: Status 404 returned error can't find the container with id 0ad92fe2832b1015d20c51a60cd9d0813a85d4c1e317da79f0ee260b0385671f
Sep 16 20:04:17.711530 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:17.711500    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:04:18.390133 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:18.390054    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zj9p/must-gather-4kqn5" event={"ID":"b7184cf4-b9cd-4d27-80ba-5e412a3b623a","Type":"ContainerStarted","Data":"0ad92fe2832b1015d20c51a60cd9d0813a85d4c1e317da79f0ee260b0385671f"}
Sep 16 20:04:19.400720 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:19.399817    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zj9p/must-gather-4kqn5" event={"ID":"b7184cf4-b9cd-4d27-80ba-5e412a3b623a","Type":"ContainerStarted","Data":"ed935b5f96a028b2be2c4fa45daa49a83954af9e458329935a0a9a1b7e7f27f9"}
Sep 16 20:04:19.400720 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:19.399880    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zj9p/must-gather-4kqn5" event={"ID":"b7184cf4-b9cd-4d27-80ba-5e412a3b623a","Type":"ContainerStarted","Data":"9a805e635083bb794b42e3a3e5e440093a7a70fb9e59e9723f4207072d9b058f"}
Sep 16 20:04:19.416422 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:19.416362    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9zj9p/must-gather-4kqn5" podStartSLOduration=1.442428815 podStartE2EDuration="2.416343411s" totalImagesPullingTime="973.914596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:04:19.415674949 +0000 UTC m=+4717.246283507" watchObservedRunningTime="2026-09-16 20:04:19.416343411 +0000 UTC m=+4717.246951956"
Sep 16 20:04:20.260650 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:20.260623    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bdq8w_5f0c80db-ca40-4eec-b140-a8e43262ed47/global-pull-secret-syncer/0.log"
Sep 16 20:04:20.382779 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:20.382743    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log"
Sep 16 20:04:20.572706 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:20.572647    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-173.ec2.internal_a86b7e4f585f41568b1e18c5a2fe21b4/haproxy/0.log"
Sep 16 20:04:22.749747 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:22.749701    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:22.821495 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:22.821461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:25.946598 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:25.945396    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-nhjkq_ed9ecc5c-c5f2-46bd-96ff-c664fae2347c/authorino/0.log"
Sep 16 20:04:26.053985 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:26.053935    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-nqdvt_0e1df62f-f49b-4f8e-bdf4-0502e635115d/kuadrant-console-plugin/0.log"
Sep 16 20:04:27.648281 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:27.648249    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_636fd7d4-7686-4ea8-8d57-a534faa3c8ff/alertmanager/0.log"
Sep 16 20:04:27.678412 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:27.678349    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_636fd7d4-7686-4ea8-8d57-a534faa3c8ff/config-reloader/0.log"
Sep 16 20:04:27.708045 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:27.708008    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_636fd7d4-7686-4ea8-8d57-a534faa3c8ff/kube-rbac-proxy-web/0.log"
Sep 16 20:04:27.741446 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:27.741412    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_636fd7d4-7686-4ea8-8d57-a534faa3c8ff/kube-rbac-proxy/0.log"
Sep 16 20:04:27.770410 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:27.770375    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_636fd7d4-7686-4ea8-8d57-a534faa3c8ff/kube-rbac-proxy-metric/0.log"
Sep 16 20:04:27.797890 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:27.797858    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_636fd7d4-7686-4ea8-8d57-a534faa3c8ff/prom-label-proxy/0.log"
Sep 16 20:04:27.834632 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:27.834553    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_636fd7d4-7686-4ea8-8d57-a534faa3c8ff/init-config-reloader/0.log"
Sep 16 20:04:28.061699 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:28.061667    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-b47cd6d9b-bxz77_421dee84-b8ca-4684-bc2a-5ef7b6b3adf8/metrics-server/0.log"
Sep 16 20:04:28.246396 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:28.246353    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmg92_f4150c27-bace-4fce-b004-8fb46d002878/node-exporter/0.log"
Sep 16 20:04:28.296286 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:28.296250    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmg92_f4150c27-bace-4fce-b004-8fb46d002878/kube-rbac-proxy/0.log"
Sep 16 20:04:28.343484 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:28.343401    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmg92_f4150c27-bace-4fce-b004-8fb46d002878/init-textfile/0.log"
Sep 16 20:04:28.977667 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:28.977634    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68f4d884dc-fxk4v_be1bfa15-8808-4400-b954-e13985b00908/telemeter-client/0.log"
Sep 16 20:04:29.008726 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:29.008690    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68f4d884dc-fxk4v_be1bfa15-8808-4400-b954-e13985b00908/reload/0.log"
Sep 16 20:04:29.040230 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:29.040193    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68f4d884dc-fxk4v_be1bfa15-8808-4400-b954-e13985b00908/kube-rbac-proxy/0.log"
Sep 16 20:04:29.077065 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:29.077031    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd4d644d6-42989_306b52a5-f1ec-4522-bd1d-c5fb0430f230/thanos-query/0.log"
Sep 16 20:04:29.105514 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:29.105482    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd4d644d6-42989_306b52a5-f1ec-4522-bd1d-c5fb0430f230/kube-rbac-proxy-web/0.log"
Sep 16 20:04:29.133467 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:29.133420    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd4d644d6-42989_306b52a5-f1ec-4522-bd1d-c5fb0430f230/kube-rbac-proxy/0.log"
Sep 16 20:04:29.159714 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:29.159683    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd4d644d6-42989_306b52a5-f1ec-4522-bd1d-c5fb0430f230/prom-label-proxy/0.log"
Sep 16 20:04:29.187868 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:29.187821    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd4d644d6-42989_306b52a5-f1ec-4522-bd1d-c5fb0430f230/kube-rbac-proxy-rules/0.log"
Sep 16 20:04:29.215988 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:29.215955    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd4d644d6-42989_306b52a5-f1ec-4522-bd1d-c5fb0430f230/kube-rbac-proxy-metrics/0.log"
Sep 16 20:04:30.279449 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:30.279417    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-mt6zb_bf969786-dd12-4361-97db-549dec34aa9e/networking-console-plugin/0.log"
Sep 16 20:04:31.057331 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.057282    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"]
Sep 16 20:04:31.063700 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.063654    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.070654 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.070616    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"]
Sep 16 20:04:31.158364 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.158066    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ed25f18-7faf-43b2-b3aa-24983431f379-sys\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.158364 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.158125    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4ed25f18-7faf-43b2-b3aa-24983431f379-proc\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.158364 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.158225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4ed25f18-7faf-43b2-b3aa-24983431f379-podres\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.158364 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.158256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf444\" (UniqueName: \"kubernetes.io/projected/4ed25f18-7faf-43b2-b3aa-24983431f379-kube-api-access-wf444\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.158364 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.158286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ed25f18-7faf-43b2-b3aa-24983431f379-lib-modules\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.259918 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.259868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4ed25f18-7faf-43b2-b3aa-24983431f379-podres\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.260117 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.259932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf444\" (UniqueName: \"kubernetes.io/projected/4ed25f18-7faf-43b2-b3aa-24983431f379-kube-api-access-wf444\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.260117 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.259964    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ed25f18-7faf-43b2-b3aa-24983431f379-lib-modules\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.260117 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.260013    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ed25f18-7faf-43b2-b3aa-24983431f379-sys\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.260117 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.260040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4ed25f18-7faf-43b2-b3aa-24983431f379-proc\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.260354 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.260147    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4ed25f18-7faf-43b2-b3aa-24983431f379-proc\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.260354 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.260173    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ed25f18-7faf-43b2-b3aa-24983431f379-lib-modules\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.260354 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.260211    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ed25f18-7faf-43b2-b3aa-24983431f379-sys\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.260685 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.260654    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4ed25f18-7faf-43b2-b3aa-24983431f379-podres\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.270296 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.270252    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf444\" (UniqueName: \"kubernetes.io/projected/4ed25f18-7faf-43b2-b3aa-24983431f379-kube-api-access-wf444\") pod \"perf-node-gather-daemonset-7jphm\" (UID: \"4ed25f18-7faf-43b2-b3aa-24983431f379\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.327835 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.327745    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56d487d596-g74ht_8125d5c6-aa86-430f-a13d-f3729eec9839/console/0.log"
Sep 16 20:04:31.378395 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.378355    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:31.672469 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:31.672435    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"]
Sep 16 20:04:31.674392 ip-10-0-142-173 kubenswrapper[2582]: W0916 20:04:31.674357    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4ed25f18_7faf_43b2_b3aa_24983431f379.slice/crio-820959608cc61831ce118928404d9bc4cc3ff4ba6b4f56966148e26f0cd4ea28 WatchSource:0}: Error finding container 820959608cc61831ce118928404d9bc4cc3ff4ba6b4f56966148e26f0cd4ea28: Status 404 returned error can't find the container with id 820959608cc61831ce118928404d9bc4cc3ff4ba6b4f56966148e26f0cd4ea28
Sep 16 20:04:32.488246 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:32.488202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm" event={"ID":"4ed25f18-7faf-43b2-b3aa-24983431f379","Type":"ContainerStarted","Data":"7450ccf3e0e77fd8a033e51cb844a99df05c228cbc50ff8270942adb58ebf78b"}
Sep 16 20:04:32.488712 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:32.488268    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm" event={"ID":"4ed25f18-7faf-43b2-b3aa-24983431f379","Type":"ContainerStarted","Data":"820959608cc61831ce118928404d9bc4cc3ff4ba6b4f56966148e26f0cd4ea28"}
Sep 16 20:04:32.488712 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:32.488382    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:32.505522 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:32.505454    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm" podStartSLOduration=1.5054281569999999 podStartE2EDuration="1.505428157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:04:32.504246119 +0000 UTC m=+4730.334854665" watchObservedRunningTime="2026-09-16 20:04:32.505428157 +0000 UTC m=+4730.336036703"
Sep 16 20:04:33.146053 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:33.146016    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tqdhb_7df2f4be-0543-439c-8a7a-2235ad96dc42/dns/0.log"
Sep 16 20:04:33.170880 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:33.170840    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tqdhb_7df2f4be-0543-439c-8a7a-2235ad96dc42/kube-rbac-proxy/0.log"
Sep 16 20:04:33.267346 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:33.267307    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-stzn8_36f66f67-8c13-4880-9052-e0624f617592/dns-node-resolver/0.log"
Sep 16 20:04:33.851780 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:33.851731    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-99b67cb9-kcqjd_f41b655b-8c93-4675-ae6b-6ac4cfebbf58/registry/0.log"
Sep 16 20:04:33.939732 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:33.939671    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z9b6q_9b5b5c2b-18f2-4855-bb08-5173767444bf/node-ca/0.log"
Sep 16 20:04:35.727685 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:35.727648    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tnknt_bf57ca8b-8820-478b-958a-b671cb621908/serve-healthcheck-canary/0.log"
Sep 16 20:04:36.336726 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:36.336690    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tbq7b_0dff464f-883c-481c-8560-d49055bc580d/kube-rbac-proxy/0.log"
Sep 16 20:04:36.362021 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:36.361996    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tbq7b_0dff464f-883c-481c-8560-d49055bc580d/exporter/0.log"
Sep 16 20:04:36.388974 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:36.388931    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tbq7b_0dff464f-883c-481c-8560-d49055bc580d/extractor/0.log"
Sep 16 20:04:38.504838 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:38.504793    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-7jphm"
Sep 16 20:04:39.518506 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:39.518466    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c6db7499f-7b9dh_fa242294-121a-44b3-84d6-ffbaae0b1472/manager/0.log"
Sep 16 20:04:39.597362 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:39.597326    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-wk7f5_f7221fc7-908b-43bb-843f-a9ca32915b4b/openshift-lws-operator/0.log"
Sep 16 20:04:47.625725 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:47.625691    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:47.726290 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:47.726257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-glsm5_6d4ca46b-f380-4a32-8163-5cdd8b6f9ec9/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:47.730925 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:47.730895    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-cdf2k_6357abcd-9ba6-4656-b180-15ac6a0cec77/migrator/0.log"
Sep 16 20:04:47.754776 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:47.754749    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-cdf2k_6357abcd-9ba6-4656-b180-15ac6a0cec77/graceful-termination/0.log"
Sep 16 20:04:49.204240 ip-10-0-142-173 kubenswrapper[2582]: I0916 20:04:49.204208    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4tbdg_091a5344-37ad-4181-9ef5-018af0208dc7/kube-multus/0.log"