Sep 17 17:57:07.892791 ip-10-0-132-0 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:57:11.662717 ip-10-0-132-0 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:57:11.662717 ip-10-0-132-0 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:57:11.662717 ip-10-0-132-0 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:57:11.662717 ip-10-0-132-0 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:57:11.662717 ip-10-0-132-0 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:57:11.663920 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.663785    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:57:11.667562 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667548    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:57:11.667597 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667563    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:57:11.667597 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667567    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:57:11.667597 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667570    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:57:11.667597 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667574    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:57:11.667597 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667577    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:57:11.667597 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667580    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:57:11.667597 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667583    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:57:11.667597 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667586    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:57:11.667597 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667590    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:57:11.667597 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667593    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:57:11.667597 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667596    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:57:11.667597 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667599    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:57:11.667597 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667601    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667604    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667609    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667612    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667615    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667618    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667620    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667623    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667625    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667627    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667630    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667632    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667635    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667637    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667640    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667642    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667645    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667647    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667650    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667652    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:57:11.667902 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667656    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667658    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667661    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667663    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667665    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667668    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667670    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667672    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667674    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667677    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667679    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667682    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667685    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667687    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667690    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667693    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667696    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667699    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667701    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667704    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:57:11.668360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667706    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667710    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667714    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667716    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667719    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667722    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667724    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667726    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667729    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667731    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667734    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667737    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667739    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667748    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667751    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667753    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667755    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667758    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:57:11.668842 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667760    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667765    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667768    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667771    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667774    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667777    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667780    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667783    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667786    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667789    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667791    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667794    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667796    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667798    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667801    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667804    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667806    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667809    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667811    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:57:11.669259 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667813    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667816    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667818    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667820    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667823    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667825    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667827    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667830    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.667832    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668600    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668607    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668609    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668612    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668615    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668618    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668620    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668623    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668625    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668628    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668631    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668644    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:57:11.669712 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668648    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668651    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668654    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668656    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668659    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668662    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668664    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668666    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668669    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668671    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668674    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668676    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668679    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668681    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668683    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668686    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668688    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668690    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668693    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668695    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:57:11.670258 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668697    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668701    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668704    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668707    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668710    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668713    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668718    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668720    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668723    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668726    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668729    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668731    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668733    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668737    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668739    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668741    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668744    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668747    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:57:11.670754 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668749    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668751    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668754    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668756    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668759    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668761    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668763    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668766    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668768    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668770    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668773    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668775    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668778    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668780    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668783    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668787    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668790    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668792    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668794    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668797    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:57:11.671183 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668800    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668803    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668806    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668809    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668812    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668815    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668817    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668819    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668822    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668824    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668827    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668829    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668831    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668834    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668836    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668838    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668841    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668843    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668846    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:57:11.671667 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668848    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668851    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668853    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668856    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668858    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668861    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668863    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668865    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668867    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.668870    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669236    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669241    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669244    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669247    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669251    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669253    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669258    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669261    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669263    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669266    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:57:11.672160 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669269    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669272    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669275    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669277    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669280    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669282    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669285    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669287    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669290    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669292    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669294    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669297    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669300    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669302    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669304    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669307    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669309    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669313    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669316    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:57:11.672640 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669319    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669321    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669324    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669327    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669330    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669333    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669336    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669338    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669341    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669344    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669347    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669351    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669353    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669356    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669358    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669361    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669363    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669365    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669368    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:57:11.673083 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669370    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669372    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669375    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669377    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669379    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669382    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669385    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669387    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669390    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669393    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669395    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669397    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669400    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669402    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669405    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669407    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669409    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669412    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669415    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669418    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:57:11.673551 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669421    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669423    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669425    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669428    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669430    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669432    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669435    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669437    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669439    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669441    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669445    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669448    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669450    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669452    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669454    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669457    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669460    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669462    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669464    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669467    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:57:11.674024 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669469    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669473    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669476    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669479    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669482    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669485    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669488    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669490    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669493    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669496    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.669499    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669579    2574 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669609    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669614    2574 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669618    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669625    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669629    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669632    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669636    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669639    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669642    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:57:11.674558 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669645    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669648    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669651    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669654    2574 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669656    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669659    2574 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669662    2574 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669665    2574 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669670    2574 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669672    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669675    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669678    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669681    2574 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669684    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669687    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669690    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669693    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669696    2574 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669699    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669701    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669704    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669707    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669711    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669714    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669717    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:57:11.675057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669720    2574 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669723    2574 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669726    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669729    2574 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669732    2574 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669735    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669738    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669740    2574 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669744    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669746    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669750    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669753    2574 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669755    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669758    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669761    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669763    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669766    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669769    2574 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669771    2574 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669775    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669777    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669780    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669783    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669786    2574 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669789    2574 flags.go:64] FLAG: --help="false"
Sep 17 17:57:11.675645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669792    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-132-0.ec2.internal"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669795    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669797    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669800    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669803    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669807    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669809    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669812    2574 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669814    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669817    2574 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669820    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669823    2574 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669825    2574 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669828    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669831    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669834    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669836    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669839    2574 flags.go:64] FLAG: --lock-file=""
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669842    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669844    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669847    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669852    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669855    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669857    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:57:11.676251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669860    2574 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669863    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669866    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669869    2574 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669871    2574 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669875    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669878    2574 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669882    2574 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669885    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669888    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669890    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669893    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669896    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669898    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669902    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669909    2574 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669912    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669915    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669917    2574 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669920    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669925    2574 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669928    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669930    2574 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669933    2574 flags.go:64] FLAG: --port="10250"
Sep 17 17:57:11.676910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669936    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669938    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fa1aa1be9b161973"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669941    2574 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669944    2574 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669946    2574 flags.go:64] FLAG: --register-node="true"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669949    2574 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669952    2574 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669958    2574 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669961    2574 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669964    2574 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669967    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669970    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669972    2574 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669979    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669982    2574 flags.go:64] FLAG: --runonce="false"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669985    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669988    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669991    2574 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669994    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669996    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.669999    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670002    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670005    2574 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670011    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670014    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:57:11.677905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670017    2574 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670020    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670023    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670026    2574 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670028    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670033    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670036    2574 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670039    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670043    2574 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670045    2574 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670048    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670050    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670053    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670056    2574 flags.go:64] FLAG: --v="2"
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670061    2574 flags.go:64] FLAG: --version="false"
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670064    2574 flags.go:64] FLAG: --vmodule=""
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670068    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670071    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670181    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670185    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670189    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670191    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670194    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:57:11.678703 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670197    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670199    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670210    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670214    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670216    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670219    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670222    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670224    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670228    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670230    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670233    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670235    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670237    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670240    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670242    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670245    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670247    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670250    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670252    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670255    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670257    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:57:11.679249 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670260    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670262    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670266    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670268    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670271    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670274    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670276    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670279    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670284    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670288    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670291    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670295    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670297    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670300    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670302    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670305    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670307    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670310    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670312    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:57:11.679772 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670314    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670318    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670321    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670323    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670325    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670327    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670330    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670332    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670335    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670337    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670339    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670342    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670344    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670347    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670349    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670353    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670355    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670358    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670360    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:57:11.680222 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670363    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670365    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670369    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670371    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670374    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670376    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670379    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670381    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670384    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670386    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670388    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670391    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670393    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670396    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670398    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670401    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670404    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670406    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670409    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:57:11.680690 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670411    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670414    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670416    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670418    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670421    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670425    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670428    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670431    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670433    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670437    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670439    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670442    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670445    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670447    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670450    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:57:11.681151 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.670454    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.670962    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.678802    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.678817    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.678835    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.678841    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678893    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678898    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678901    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678904    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678907    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678910    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678913    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678915    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678918    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:57:11.681508 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678921    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678923    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678926    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678929    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678931    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678934    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678937    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678939    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678941    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678944    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678947    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678949    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678951    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678954    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678956    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678959    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678961    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678964    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678967    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:57:11.681932 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678969    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678974    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678977    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678980    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678983    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678985    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678988    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678990    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678993    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678995    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.678998    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679000    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679003    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679005    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679008    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679011    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679015    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679018    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679021    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:57:11.682400 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679023    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679026    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679028    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679031    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679033    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679036    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679038    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679041    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679043    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679046    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679049    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679051    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679054    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679057    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679061    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679065    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679068    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679071    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679074    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:57:11.682893 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679077    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679080    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679082    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679086    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679088    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679090    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679093    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679095    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679098    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679100    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679103    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679105    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679108    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679111    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679114    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679116    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679119    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679121    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679124    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679127    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:57:11.683360 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679129    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679132    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679134    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679137    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679139    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679142    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679145    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679147    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679150    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679152    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679155    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679158    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679160    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.679165    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679270    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:57:11.683852 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679275    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679277    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679280    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679283    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679285    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679288    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679290    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679293    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679295    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679298    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679301    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679304    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679306    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679309    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679311    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679313    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679316    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679318    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679321    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679323    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:57:11.684204 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679327    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679331    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679334    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679337    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679340    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679343    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679345    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679348    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679350    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679353    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679355    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679358    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679361    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679363    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679366    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679368    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679371    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679374    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679376    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679379    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:57:11.684821 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679381    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679384    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679387    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679389    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679393    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679395    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679398    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679401    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679403    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679406    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679408    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679410    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679413    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679415    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679418    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679420    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679422    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679425    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679427    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679430    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:57:11.685292 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679432    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679435    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679437    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679439    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679442    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679444    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679446    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679449    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679451    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679453    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679456    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679458    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679461    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679463    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679465    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679468    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679470    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679473    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679475    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:57:11.685790 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679478    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679480    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679483    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679485    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679488    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679490    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679493    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679496    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679499    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679503    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679506    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679530    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679534    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679537    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679540    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679542    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679545    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679548    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:57:11.686227 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:11.679550    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:57:11.686668 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.679555    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:57:11.686668 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.679663    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:57:11.686668 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.681579    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:57:11.686668 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.682560    2574 server.go:1019] "Starting client certificate rotation"
Sep 17 17:57:11.686668 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.682665    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:57:11.686668 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.682702    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:57:11.686668 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.685455    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:57:11.686668 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.686537    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:57:11.690371 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.690353    2574 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:57:11.692880 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.692867    2574 log.go:25] "Validated CRI v1 image API"
Sep 17 17:57:11.693681 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.693665    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:57:11.696462 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.696445    2574 fs.go:135] Filesystem UUIDs: map[07fd2db5-fb80-4cdf-960c-0e23aa3548d2:/dev/nvme0n1p4 75219070-dd3e-4449-b214-7e519ecd02f1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 17 17:57:11.696534 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.696462    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:57:11.701376 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.701284    2574 manager.go:216] Machine: {Timestamp:2026-09-17 17:57:11.700839454 +0000 UTC m=+2.001661774 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200351 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aea6379cc099e1c05d899d53bc540 SystemUUID:ec2aea63-79cc-099e-1c05-d899d53bc540 BootID:e8a04cfb-8f40-4c88-afb5-be3836ff35d1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e8:f1:be:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e8:f1:be:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:04:88:c6:8b:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:57:11.701376 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.701373    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:57:11.701484 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.701437    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:57:11.703081 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.703062    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:57:11.703275 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.703083    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-0.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:57:11.705029 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.704873    2574 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:57:11.705098 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.705034    2574 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:57:11.705098 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.705051    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:57:11.705098 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.705063    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:57:11.705236 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.705135    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:57:11.708097 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.708086    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:57:11.708209 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.708199    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:57:11.708245 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.708231    2574 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:57:11.708245 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.708242    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:57:11.708331 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.708264    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:57:11.708331 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.708275    2574 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:57:11.708331 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.708285    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:57:11.708622 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.708609    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:57:11.708685 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.708640    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:57:11.711000 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.710980    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:57:11.713151 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.713138    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:57:11.713204 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.713168    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:57:11.714033 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714012    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:57:11.714754 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714743    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:57:11.714790 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714758    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:57:11.714790 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714764    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:57:11.714790 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714770    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:57:11.714790 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714778    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:57:11.714790 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714783    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:57:11.714912 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714797    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:57:11.714912 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714802    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:57:11.714912 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714809    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:57:11.714912 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714814    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:57:11.714912 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714822    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:57:11.714912 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714827    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:57:11.714912 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714849    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:57:11.714912 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.714855    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:57:11.716696 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.716681    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-0.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:57:11.717287 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:11.717268    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-0.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:57:11.717320 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:11.717273    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:57:11.718401 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.718391    2574 server.go:1268] "Started kubelet"
Sep 17 17:57:11.718447 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.718433    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:57:11.718527 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.718477    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:57:11.718572 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.718554    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:57:11.719158 ip-10-0-132-0 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:57:11.719731 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.719666    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:57:11.720104 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.720092    2574 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:57:11.722342 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:11.722315    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:57:11.725391 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.725378    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:57:11.725463 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.725388    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:57:11.725557 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:11.725536    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-0.ec2.internal\" not found"
Sep 17 17:57:11.725941 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.725914    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:57:11.725941 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.725933    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:57:11.726049 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.726010    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:57:11.726049 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.726026    2574 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:57:11.726049 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.726032    2574 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:57:11.726312 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.726209    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:57:11.726312 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.726312    2574 factory.go:55] Registering systemd factory
Sep 17 17:57:11.726446 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.726324    2574 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:57:11.726575 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.726559    2574 factory.go:153] Registering CRI-O factory
Sep 17 17:57:11.726575 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.726576    2574 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:57:11.726765 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.726596    2574 factory.go:103] Registering Raw factory
Sep 17 17:57:11.726765 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.726609    2574 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:57:11.727061 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.727035    2574 manager.go:349] Starting recovery of all containers
Sep 17 17:57:11.746621 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:11.746589    2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-0.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:57:11.746734 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:11.746718    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:57:11.754661 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.754644    2574 manager.go:354] Recovery completed
Sep 17 17:57:11.758542 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.758527    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:57:11.760701 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.760675    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:57:11.760768 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.760713    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:57:11.760768 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.760722    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:57:11.761236 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.761220    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:57:11.761236 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.761231    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:57:11.761470 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.761234    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:57:11.761470 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.761302    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:57:11.762899 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.762882    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:57:11.762967 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.762916    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:57:11.762967 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.762929    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:57:11.763399 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.763386    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8c6kc"
Sep 17 17:57:11.766471 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:11.765668    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-0.ec2.internal.18d62d692f8442a3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-0.ec2.internal,UID:ip-10-0-132-0.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-0.ec2.internal,},FirstTimestamp:2026-09-17 17:57:11.718363811 +0000 UTC m=+2.019186124,LastTimestamp:2026-09-17 17:57:11.718363811 +0000 UTC m=+2.019186124,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-0.ec2.internal,}"
Sep 17 17:57:11.767245 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.767231    2574 policy_none.go:49] "None policy: Start"
Sep 17 17:57:11.767315 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.767253    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:57:11.767315 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.767267    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:57:11.768453 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.768440    2574 policy_none.go:47] "Start"
Sep 17 17:57:11.776539 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.776424    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8c6kc"
Sep 17 17:57:11.779714 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:11.779655    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-0.ec2.internal.18d62d69320a45c4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-0.ec2.internal,UID:ip-10-0-132-0.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-0.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-0.ec2.internal,},FirstTimestamp:2026-09-17 17:57:11.760700868 +0000 UTC m=+2.061523184,LastTimestamp:2026-09-17 17:57:11.760700868 +0000 UTC m=+2.061523184,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-0.ec2.internal,}"
Sep 17 17:57:11.814850 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.802780    2574 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:57:11.814850 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:11.802816    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:57:11.814850 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.802825    2574 server.go:87] "Starting device plugin registration server"
Sep 17 17:57:11.814850 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.803047    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:57:11.814850 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.803067    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:57:11.814850 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.803137    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:57:11.814850 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.803210    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:57:11.814850 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.803219    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:57:11.814850 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:11.803831    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:57:11.814850 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:11.803874    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-0.ec2.internal\" not found"
Sep 17 17:57:11.903469 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.903449    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:57:11.904467 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.904453    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:57:11.904560 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.904478    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:57:11.904560 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.904489    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:57:11.904560 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.904530    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-0.ec2.internal"
Sep 17 17:57:11.916361 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.916323    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-0.ec2.internal"
Sep 17 17:57:11.916361 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:11.916342    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-0.ec2.internal\": node \"ip-10-0-132-0.ec2.internal\" not found"
Sep 17 17:57:11.930359 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.930340    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:57:11.931638 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.931626    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:57:11.931703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.931667    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:57:11.931703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.931685    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:57:11.931798 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:11.931723    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:57:11.935123 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:11.935106    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:57:11.939249 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:11.939235    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-0.ec2.internal\" not found"
Sep 17 17:57:12.032463 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.032438    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-0.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal"]
Sep 17 17:57:12.032623 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.032507    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:57:12.034043 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.034027    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:57:12.034153 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.034061    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:57:12.034153 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.034077    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:57:12.035416 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.035401    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:57:12.035536 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.035507    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.035582 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.035547    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:57:12.036322 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.036303    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:57:12.036322 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.036315    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:57:12.036500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.036333    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:57:12.036500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.036340    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:57:12.036500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.036346    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:57:12.036500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.036353    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:57:12.037821 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.037805    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.037888 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.037831    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:57:12.038706 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.038689    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:57:12.038786 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.038715    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:57:12.038786 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.038725    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:57:12.039446 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.039432    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-0.ec2.internal\" not found"
Sep 17 17:57:12.052987 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.052966    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-0.ec2.internal\" not found" node="ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.055976 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.055961    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-0.ec2.internal\" not found" node="ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.129067 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.129042    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef7d3c93d84d7fbd4bfd4bfbc5d80220-config\") pod \"kube-apiserver-proxy-ip-10-0-132-0.ec2.internal\" (UID: \"ef7d3c93d84d7fbd4bfd4bfbc5d80220\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.129067 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.129065    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f58a43d59107368b45168fc33760fb7b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal\" (UID: \"f58a43d59107368b45168fc33760fb7b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.129191 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.129085    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f58a43d59107368b45168fc33760fb7b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal\" (UID: \"f58a43d59107368b45168fc33760fb7b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.139729 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.139707    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-0.ec2.internal\" not found"
Sep 17 17:57:12.229227 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.229166    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef7d3c93d84d7fbd4bfd4bfbc5d80220-config\") pod \"kube-apiserver-proxy-ip-10-0-132-0.ec2.internal\" (UID: \"ef7d3c93d84d7fbd4bfd4bfbc5d80220\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.229227 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.229191    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f58a43d59107368b45168fc33760fb7b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal\" (UID: \"f58a43d59107368b45168fc33760fb7b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.229227 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.229208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f58a43d59107368b45168fc33760fb7b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal\" (UID: \"f58a43d59107368b45168fc33760fb7b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.229380 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.229242    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f58a43d59107368b45168fc33760fb7b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal\" (UID: \"f58a43d59107368b45168fc33760fb7b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.229380 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.229249    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef7d3c93d84d7fbd4bfd4bfbc5d80220-config\") pod \"kube-apiserver-proxy-ip-10-0-132-0.ec2.internal\" (UID: \"ef7d3c93d84d7fbd4bfd4bfbc5d80220\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.229380 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.229273    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f58a43d59107368b45168fc33760fb7b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal\" (UID: \"f58a43d59107368b45168fc33760fb7b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.240298 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.240276    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-0.ec2.internal\" not found"
Sep 17 17:57:12.340774 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.340739    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-0.ec2.internal\" not found"
Sep 17 17:57:12.356933 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.356903    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.360044 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.360028    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.441788 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.441759    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-0.ec2.internal\" not found"
Sep 17 17:57:12.542364 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.542312    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-0.ec2.internal\" not found"
Sep 17 17:57:12.642694 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.642662    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-0.ec2.internal\" not found"
Sep 17 17:57:12.643392 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.643374    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:57:12.682435 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.682416    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:57:12.682868 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.682541    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:57:12.682868 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.682582    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:57:12.708742 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.708717    2574 apiserver.go:52] "Watching apiserver"
Sep 17 17:57:12.725881 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.725858    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.725959 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.725889    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:57:12.733285 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.733264    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:57:12.739811 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.739790    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:57:12.739977 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.739959    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal"
Sep 17 17:57:12.740856 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.740836    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:57:12.741137 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.741119    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-kb4ks","kube-system/global-pull-secret-syncer-4kg4f","kube-system/konnectivity-agent-6jhpj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9","openshift-dns/node-resolver-4mlb8","openshift-multus/multus-additional-cni-plugins-kp6mp","openshift-multus/network-metrics-daemon-xjh98","openshift-network-operator/iptables-alerter-mm6bn","openshift-ovn-kubernetes/ovnkube-node-j2v2z","openshift-cluster-node-tuning-operator/tuned-vb6r7","openshift-image-registry/node-ca-4vwlx","openshift-multus/multus-xfm5k"]
Sep 17 17:57:12.742596 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.742581    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.743770 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.743746    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4mlb8"
Sep 17 17:57:12.744795 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.744781    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:57:12.744938 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.744925    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.746039 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.746022    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:12.746132 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.746087    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:12.747000 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.746984    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:57:12.747091 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.747075    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:12.747167 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.747150    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:12.747851 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.747817    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:57:12.747954 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.747940    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f2xmg\""
Sep 17 17:57:12.748161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.748146    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:57:12.748273 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.748254    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:57:12.748273 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.748254    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:57:12.748379 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.748332    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:57:12.748485 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.748472    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h7lph\""
Sep 17 17:57:12.748630 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.748617    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-knkbz\""
Sep 17 17:57:12.748700 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.748673    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6jhpj"
Sep 17 17:57:12.750120 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.750107    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.751366 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.751351    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:12.751430 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.751403    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:12.752566 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.752555    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mm6bn"
Sep 17 17:57:12.753843 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.753829    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.755133 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.755117    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4vwlx"
Sep 17 17:57:12.756238 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.756218    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:57:12.756382 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.756357    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-0.ec2.internal"]
Sep 17 17:57:12.756473 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.756461    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.758424 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.758408    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:57:12.760779 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.760759    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2ktng\""
Sep 17 17:57:12.760875 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.760779    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:57:12.760875 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.760787    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:57:12.760963 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.760924    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:57:12.761057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.761042    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:57:12.761107 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.761080    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:57:12.761333 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.761310    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lwgbm\""
Sep 17 17:57:12.761439 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.761423    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:57:12.761439 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.761433    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:57:12.761550 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.761458    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-srmt7\""
Sep 17 17:57:12.762291 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.761823    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:57:12.762291 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.761855    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:57:12.762291 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.761423    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:57:12.762291 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.762030    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:57:12.762291 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.762140    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:57:12.762291 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.762187    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gvbr4\""
Sep 17 17:57:12.762611 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.762427    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:57:12.762611 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.762578    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:57:12.762701 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.762610    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zch5c\""
Sep 17 17:57:12.762907 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.762890    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:57:12.762959 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.762917    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:57:12.762959 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.762924    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:57:12.762959 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.762942    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:57:12.763313 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.763297    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gxw8w\""
Sep 17 17:57:12.767057 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.767038    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:57:12.767790 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.767778    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal"]
Sep 17 17:57:12.768109 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.768097    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:57:12.781135 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.781095    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:52:11 +0000 UTC" deadline="2028-07-04 08:58:29.1986569 +0000 UTC"
Sep 17 17:57:12.781135 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.781135    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15735h1m16.417524339s"
Sep 17 17:57:12.811036 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.811020    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l7s42"
Sep 17 17:57:12.820418 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.820400    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l7s42"
Sep 17 17:57:12.827651 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.827633    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:57:12.832184 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832167    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-kubernetes\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.832264 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832189    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-kubelet\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.832264 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832205    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53270d0a-dc99-4f59-a2a3-2086917c92f2-ovnkube-config\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.832264 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832221    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8fb6002-f122-4cc8-a73b-3d1e45addd6e-serviceca\") pod \"node-ca-4vwlx\" (UID: \"a8fb6002-f122-4cc8-a73b-3d1e45addd6e\") " pod="openshift-image-registry/node-ca-4vwlx"
Sep 17 17:57:12.832264 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832244    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm6h8\" (UniqueName: \"kubernetes.io/projected/a8fb6002-f122-4cc8-a73b-3d1e45addd6e-kube-api-access-qm6h8\") pod \"node-ca-4vwlx\" (UID: \"a8fb6002-f122-4cc8-a73b-3d1e45addd6e\") " pod="openshift-image-registry/node-ca-4vwlx"
Sep 17 17:57:12.832408 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-multus-socket-dir-parent\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.832408 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832290    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pp7w\" (UniqueName: \"kubernetes.io/projected/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-kube-api-access-7pp7w\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.832408 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832323    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/229dfe1c-397d-48b4-b1d4-e3c5e6429339-agent-certs\") pod \"konnectivity-agent-6jhpj\" (UID: \"229dfe1c-397d-48b4-b1d4-e3c5e6429339\") " pod="kube-system/konnectivity-agent-6jhpj"
Sep 17 17:57:12.832408 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832345    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvt5r\" (UniqueName: \"kubernetes.io/projected/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-kube-api-access-nvt5r\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.832408 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832362    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs\") pod \"network-metrics-daemon-xjh98\" (UID: \"3ee045a7-cfc1-40b2-87a8-65bedec89f66\") " pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:12.832408 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832394    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8m9z\" (UniqueName: \"kubernetes.io/projected/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-kube-api-access-s8m9z\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.832675 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832418    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-systemd-units\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.832675 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832434    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-run-openvswitch\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.832675 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.832675 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832474    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6126893-24bd-4045-a954-7dd893aff7e8-kubelet-config\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:12.832675 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832488    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-modprobe-d\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.832675 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832501    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-node-log\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.832675 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.832675 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832565    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-run-k8s-cni-cncf-io\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.832675 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832595    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-multus-conf-dir\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.832675 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832616    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-222v9\" (UniqueName: \"kubernetes.io/projected/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-kube-api-access-222v9\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.832675 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832652    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-log-socket\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.832675 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832673    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvqwj\" (UniqueName: \"kubernetes.io/projected/3ee045a7-cfc1-40b2-87a8-65bedec89f66-kube-api-access-hvqwj\") pod \"network-metrics-daemon-xjh98\" (UID: \"3ee045a7-cfc1-40b2-87a8-65bedec89f66\") " pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832702    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-multus-cni-dir\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832730    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-var-lib-kubelet\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832754    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzhlp\" (UniqueName: \"kubernetes.io/projected/c702e515-fb08-4a73-9ac8-3295f2c18596-kube-api-access-kzhlp\") pod \"iptables-alerter-mm6bn\" (UID: \"c702e515-fb08-4a73-9ac8-3295f2c18596\") " pod="openshift-network-operator/iptables-alerter-mm6bn"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832779    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmgjk\" (UniqueName: \"kubernetes.io/projected/53270d0a-dc99-4f59-a2a3-2086917c92f2-kube-api-access-rmgjk\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832803    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832869    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c380ad-08a8-4689-b88f-f07b659e6314-tmp-dir\") pod \"node-resolver-4mlb8\" (UID: \"e2c380ad-08a8-4689-b88f-f07b659e6314\") " pod="openshift-dns/node-resolver-4mlb8"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-run-multus-certs\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832936    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-sys-fs\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832958    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-tmp\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.832981    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c702e515-fb08-4a73-9ac8-3295f2c18596-host-slash\") pod \"iptables-alerter-mm6bn\" (UID: \"c702e515-fb08-4a73-9ac8-3295f2c18596\") " pod="openshift-network-operator/iptables-alerter-mm6bn"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833005    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-etc-openvswitch\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833026    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8fb6002-f122-4cc8-a73b-3d1e45addd6e-host\") pod \"node-ca-4vwlx\" (UID: \"a8fb6002-f122-4cc8-a73b-3d1e45addd6e\") " pod="openshift-image-registry/node-ca-4vwlx"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833052    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e2c380ad-08a8-4689-b88f-f07b659e6314-hosts-file\") pod \"node-resolver-4mlb8\" (UID: \"e2c380ad-08a8-4689-b88f-f07b659e6314\") " pod="openshift-dns/node-resolver-4mlb8"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833090    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-tuned\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.833161 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833121    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-run-systemd\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.833714 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833216    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53270d0a-dc99-4f59-a2a3-2086917c92f2-ovn-node-metrics-cert\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.833714 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833252    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-cnibin\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.833714 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833326    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-multus-daemon-config\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.833714 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-device-dir\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.833714 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53270d0a-dc99-4f59-a2a3-2086917c92f2-env-overrides\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.833714 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833506    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbctq\" (UniqueName: \"kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq\") pod \"network-check-target-kb4ks\" (UID: \"0081dcb0-3253-477a-8556-da52f047f100\") " pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:12.833714 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833565    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-cni-binary-copy\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.833714 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833599    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-var-lib-cni-bin\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.833714 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833620    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-sysctl-d\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.833714 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833639    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-host\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.833714 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833661    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-run-ovn\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.833714 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833693    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-run-netns\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833730    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-etc-kubernetes\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833762    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833812    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-systemd\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833848    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c702e515-fb08-4a73-9ac8-3295f2c18596-iptables-alerter-script\") pod \"iptables-alerter-mm6bn\" (UID: \"c702e515-fb08-4a73-9ac8-3295f2c18596\") " pod="openshift-network-operator/iptables-alerter-mm6bn"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/229dfe1c-397d-48b4-b1d4-e3c5e6429339-konnectivity-ca\") pod \"konnectivity-agent-6jhpj\" (UID: \"229dfe1c-397d-48b4-b1d4-e3c5e6429339\") " pod="kube-system/konnectivity-agent-6jhpj"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833911    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-sysconfig\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833947    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-sysctl-conf\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833966    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-var-lib-kubelet\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.833987    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-run-netns\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834011    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834032    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834056    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24qh6\" (UniqueName: \"kubernetes.io/projected/e2c380ad-08a8-4689-b88f-f07b659e6314-kube-api-access-24qh6\") pod \"node-resolver-4mlb8\" (UID: \"e2c380ad-08a8-4689-b88f-f07b659e6314\") " pod="openshift-dns/node-resolver-4mlb8"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-cni-bin\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834115    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-system-cni-dir\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.834172 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834162    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-hostroot\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834192    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-run\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-var-lib-openvswitch\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834259    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-cni-netd\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53270d0a-dc99-4f59-a2a3-2086917c92f2-ovnkube-script-lib\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834315    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-system-cni-dir\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834329    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-cni-binary-copy\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834343    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-lib-modules\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834355    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-slash\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834382    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6126893-24bd-4045-a954-7dd893aff7e8-dbus\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834403    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834417    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-cnibin\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834430    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-registration-dir\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834442    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-sys\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834462    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834492    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-var-lib-cni-multus\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.834749 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.834507    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-socket-dir\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.935620 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935594    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-lib-modules\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.935620 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935621    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-slash\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.935815 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935636    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6126893-24bd-4045-a954-7dd893aff7e8-dbus\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:12.935815 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935650    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:12.935815 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-cnibin\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.935815 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935685    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-registration-dir\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.935815 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935705    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-sys\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.935815 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935707    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-slash\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.935815 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935724    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.935815 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935747    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-var-lib-cni-multus\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.935815 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935767    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-socket-dir\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.935815 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935769    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-cnibin\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.935815 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935790    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-kubernetes\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.935815 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935806    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6126893-24bd-4045-a954-7dd893aff7e8-dbus\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935852    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-kubernetes\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935864    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-var-lib-cni-multus\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935776    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-sys\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935886    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-kubelet\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935888    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-registration-dir\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935915    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53270d0a-dc99-4f59-a2a3-2086917c92f2-ovnkube-config\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935920    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-socket-dir\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.935933    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935917    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-kubelet\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.935940    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8fb6002-f122-4cc8-a73b-3d1e45addd6e-serviceca\") pod \"node-ca-4vwlx\" (UID: \"a8fb6002-f122-4cc8-a73b-3d1e45addd6e\") " pod="openshift-image-registry/node-ca-4vwlx"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.936032    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret podName:c6126893-24bd-4045-a954-7dd893aff7e8 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:13.435993926 +0000 UTC m=+3.736816247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret") pod "global-pull-secret-syncer-4kg4f" (UID: "c6126893-24bd-4045-a954-7dd893aff7e8") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936076    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6h8\" (UniqueName: \"kubernetes.io/projected/a8fb6002-f122-4cc8-a73b-3d1e45addd6e-kube-api-access-qm6h8\") pod \"node-ca-4vwlx\" (UID: \"a8fb6002-f122-4cc8-a73b-3d1e45addd6e\") " pod="openshift-image-registry/node-ca-4vwlx"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-multus-socket-dir-parent\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936134    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pp7w\" (UniqueName: \"kubernetes.io/projected/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-kube-api-access-7pp7w\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936178    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/229dfe1c-397d-48b4-b1d4-e3c5e6429339-agent-certs\") pod \"konnectivity-agent-6jhpj\" (UID: \"229dfe1c-397d-48b4-b1d4-e3c5e6429339\") " pod="kube-system/konnectivity-agent-6jhpj"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936221    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvt5r\" (UniqueName: \"kubernetes.io/projected/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-kube-api-access-nvt5r\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.936360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs\") pod \"network-metrics-daemon-xjh98\" (UID: \"3ee045a7-cfc1-40b2-87a8-65bedec89f66\") " pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8m9z\" (UniqueName: \"kubernetes.io/projected/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-kube-api-access-s8m9z\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936309    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-systemd-units\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936329    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-run-openvswitch\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936349    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936371    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6126893-24bd-4045-a954-7dd893aff7e8-kubelet-config\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936394    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-modprobe-d\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936405    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-lib-modules\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936432    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-node-log\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936457    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936485    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-systemd-units\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936533    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-run-k8s-cni-cncf-io\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936561    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-multus-conf-dir\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936575    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-run-openvswitch\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-222v9\" (UniqueName: \"kubernetes.io/projected/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-kube-api-access-222v9\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-log-socket\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvqwj\" (UniqueName: \"kubernetes.io/projected/3ee045a7-cfc1-40b2-87a8-65bedec89f66-kube-api-access-hvqwj\") pod \"network-metrics-daemon-xjh98\" (UID: \"3ee045a7-cfc1-40b2-87a8-65bedec89f66\") " pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:12.937119 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936641    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-multus-cni-dir\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936642    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936653    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-var-lib-kubelet\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzhlp\" (UniqueName: \"kubernetes.io/projected/c702e515-fb08-4a73-9ac8-3295f2c18596-kube-api-access-kzhlp\") pod \"iptables-alerter-mm6bn\" (UID: \"c702e515-fb08-4a73-9ac8-3295f2c18596\") " pod="openshift-network-operator/iptables-alerter-mm6bn"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936697    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmgjk\" (UniqueName: \"kubernetes.io/projected/53270d0a-dc99-4f59-a2a3-2086917c92f2-kube-api-access-rmgjk\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936733    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936764    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-log-socket\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936775    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c380ad-08a8-4689-b88f-f07b659e6314-tmp-dir\") pod \"node-resolver-4mlb8\" (UID: \"e2c380ad-08a8-4689-b88f-f07b659e6314\") " pod="openshift-dns/node-resolver-4mlb8"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936806    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-run-multus-certs\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936825    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-modprobe-d\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936832    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-sys-fs\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936890    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-tmp\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936913    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8fb6002-f122-4cc8-a73b-3d1e45addd6e-serviceca\") pod \"node-ca-4vwlx\" (UID: \"a8fb6002-f122-4cc8-a73b-3d1e45addd6e\") " pod="openshift-image-registry/node-ca-4vwlx"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937003    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c702e515-fb08-4a73-9ac8-3295f2c18596-host-slash\") pod \"iptables-alerter-mm6bn\" (UID: \"c702e515-fb08-4a73-9ac8-3295f2c18596\") " pod="openshift-network-operator/iptables-alerter-mm6bn"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936918    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c702e515-fb08-4a73-9ac8-3295f2c18596-host-slash\") pod \"iptables-alerter-mm6bn\" (UID: \"c702e515-fb08-4a73-9ac8-3295f2c18596\") " pod="openshift-network-operator/iptables-alerter-mm6bn"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-etc-openvswitch\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937071    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8fb6002-f122-4cc8-a73b-3d1e45addd6e-host\") pod \"node-ca-4vwlx\" (UID: \"a8fb6002-f122-4cc8-a73b-3d1e45addd6e\") " pod="openshift-image-registry/node-ca-4vwlx"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937079    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-multus-cni-dir\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.937922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e2c380ad-08a8-4689-b88f-f07b659e6314-hosts-file\") pod \"node-resolver-4mlb8\" (UID: \"e2c380ad-08a8-4689-b88f-f07b659e6314\") " pod="openshift-dns/node-resolver-4mlb8"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.936710    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-multus-socket-dir-parent\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937128    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-var-lib-kubelet\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937122    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-tuned\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937170    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-run-systemd\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937172    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-run-k8s-cni-cncf-io\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937206    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-node-log\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937240    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53270d0a-dc99-4f59-a2a3-2086917c92f2-ovn-node-metrics-cert\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937276    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-cnibin\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937304    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-multus-daemon-config\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937294    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/53270d0a-dc99-4f59-a2a3-2086917c92f2-ovnkube-config\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937331    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-device-dir\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937360    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53270d0a-dc99-4f59-a2a3-2086917c92f2-env-overrides\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937389    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbctq\" (UniqueName: \"kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq\") pod \"network-check-target-kb4ks\" (UID: \"0081dcb0-3253-477a-8556-da52f047f100\") " pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937399    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937414    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-cni-binary-copy\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937442    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-var-lib-cni-bin\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937458    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-multus-conf-dir\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.938910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937455    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-etc-openvswitch\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937462    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-sysctl-d\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937508    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-host\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937557    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-run-ovn\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.937591    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937584    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c380ad-08a8-4689-b88f-f07b659e6314-tmp-dir\") pod \"node-resolver-4mlb8\" (UID: \"e2c380ad-08a8-4689-b88f-f07b659e6314\") " pod="openshift-dns/node-resolver-4mlb8"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937630    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-run-netns\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937647    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.937660    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs podName:3ee045a7-cfc1-40b2-87a8-65bedec89f66 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:13.437631547 +0000 UTC m=+3.738454046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs") pod "network-metrics-daemon-xjh98" (UID: "3ee045a7-cfc1-40b2-87a8-65bedec89f66") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937578    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-sysctl-d\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937672    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-run-netns\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937739    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-host\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937739    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-sys-fs\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937767    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-run-systemd\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937810    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-cnibin\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937244    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-etc-kubernetes\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.939740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938301    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-multus-daemon-config\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938328    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938365    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-device-dir\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938376    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-systemd\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938378    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938423    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c702e515-fb08-4a73-9ac8-3295f2c18596-iptables-alerter-script\") pod \"iptables-alerter-mm6bn\" (UID: \"c702e515-fb08-4a73-9ac8-3295f2c18596\") " pod="openshift-network-operator/iptables-alerter-mm6bn"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938456    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/229dfe1c-397d-48b4-b1d4-e3c5e6429339-konnectivity-ca\") pod \"konnectivity-agent-6jhpj\" (UID: \"229dfe1c-397d-48b4-b1d4-e3c5e6429339\") " pod="kube-system/konnectivity-agent-6jhpj"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938476    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e2c380ad-08a8-4689-b88f-f07b659e6314-hosts-file\") pod \"node-resolver-4mlb8\" (UID: \"e2c380ad-08a8-4689-b88f-f07b659e6314\") " pod="openshift-dns/node-resolver-4mlb8"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-sysconfig\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-sysctl-conf\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938595    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-var-lib-kubelet\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938586    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6126893-24bd-4045-a954-7dd893aff7e8-kubelet-config\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-run-netns\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938677    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938716    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938740    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24qh6\" (UniqueName: \"kubernetes.io/projected/e2c380ad-08a8-4689-b88f-f07b659e6314-kube-api-access-24qh6\") pod \"node-resolver-4mlb8\" (UID: \"e2c380ad-08a8-4689-b88f-f07b659e6314\") " pod="openshift-dns/node-resolver-4mlb8"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938758    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-cni-bin\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.940500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938774    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-system-cni-dir\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938794    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-hostroot\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938813    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-run\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938844    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-var-lib-openvswitch\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-cni-netd\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938909    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/53270d0a-dc99-4f59-a2a3-2086917c92f2-env-overrides\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938918    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53270d0a-dc99-4f59-a2a3-2086917c92f2-ovnkube-script-lib\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-system-cni-dir\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.938976    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-cni-binary-copy\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939063    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-run-ovn\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.937377    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8fb6002-f122-4cc8-a73b-3d1e45addd6e-host\") pod \"node-ca-4vwlx\" (UID: \"a8fb6002-f122-4cc8-a73b-3d1e45addd6e\") " pod="openshift-image-registry/node-ca-4vwlx"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939173    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-var-lib-cni-bin\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939212    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-host-run-multus-certs\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939224    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c702e515-fb08-4a73-9ac8-3295f2c18596-iptables-alerter-script\") pod \"iptables-alerter-mm6bn\" (UID: \"c702e515-fb08-4a73-9ac8-3295f2c18596\") " pod="openshift-network-operator/iptables-alerter-mm6bn"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939266    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-etc-kubernetes\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939328    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-var-lib-kubelet\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939335    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939384    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-systemd\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.941274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939444    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-run\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939446    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-run-netns\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939386    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-sysconfig\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939491    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-cni-bin\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939544    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-cni-binary-copy\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939579    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-sysctl-conf\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939708    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939741    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-var-lib-openvswitch\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.939821    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/53270d0a-dc99-4f59-a2a3-2086917c92f2-host-cni-netd\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.940048    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/229dfe1c-397d-48b4-b1d4-e3c5e6429339-konnectivity-ca\") pod \"konnectivity-agent-6jhpj\" (UID: \"229dfe1c-397d-48b4-b1d4-e3c5e6429339\") " pod="kube-system/konnectivity-agent-6jhpj"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.940140    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.940203    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-hostroot\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.940195    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-system-cni-dir\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.940257    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-system-cni-dir\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.940345    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/53270d0a-dc99-4f59-a2a3-2086917c92f2-ovnkube-script-lib\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.940347    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-cni-binary-copy\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.941685    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-tmp\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.941906    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-etc-tuned\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.942071 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.941988    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/229dfe1c-397d-48b4-b1d4-e3c5e6429339-agent-certs\") pod \"konnectivity-agent-6jhpj\" (UID: \"229dfe1c-397d-48b4-b1d4-e3c5e6429339\") " pod="kube-system/konnectivity-agent-6jhpj"
Sep 17 17:57:12.942853 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.942836    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/53270d0a-dc99-4f59-a2a3-2086917c92f2-ovn-node-metrics-cert\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.956069 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.956042    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6h8\" (UniqueName: \"kubernetes.io/projected/a8fb6002-f122-4cc8-a73b-3d1e45addd6e-kube-api-access-qm6h8\") pod \"node-ca-4vwlx\" (UID: \"a8fb6002-f122-4cc8-a73b-3d1e45addd6e\") " pod="openshift-image-registry/node-ca-4vwlx"
Sep 17 17:57:12.956299 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.956275    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:12.956363 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.956298    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:12.956573 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.956534    2574 projected.go:196] Error preparing data for projected volume kube-api-access-tbctq for pod openshift-network-diagnostics/network-check-target-kb4ks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:12.956987 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:12.956969    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq podName:0081dcb0-3253-477a-8556-da52f047f100 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:13.456935837 +0000 UTC m=+3.757758138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tbctq" (UniqueName: "kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq") pod "network-check-target-kb4ks" (UID: "0081dcb0-3253-477a-8556-da52f047f100") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:12.957269 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.956306    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzhlp\" (UniqueName: \"kubernetes.io/projected/c702e515-fb08-4a73-9ac8-3295f2c18596-kube-api-access-kzhlp\") pod \"iptables-alerter-mm6bn\" (UID: \"c702e515-fb08-4a73-9ac8-3295f2c18596\") " pod="openshift-network-operator/iptables-alerter-mm6bn"
Sep 17 17:57:12.959632 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.959609    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8m9z\" (UniqueName: \"kubernetes.io/projected/030fbe18-1e3d-415f-a113-3e7e1ca71d1b-kube-api-access-s8m9z\") pod \"tuned-vb6r7\" (UID: \"030fbe18-1e3d-415f-a113-3e7e1ca71d1b\") " pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:12.959792 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.959773    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pp7w\" (UniqueName: \"kubernetes.io/projected/e6b5dcf7-1ca9-4379-b734-2bce67c4d84d-kube-api-access-7pp7w\") pod \"aws-ebs-csi-driver-node-dc2f9\" (UID: \"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:12.960739 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.960124    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24qh6\" (UniqueName: \"kubernetes.io/projected/e2c380ad-08a8-4689-b88f-f07b659e6314-kube-api-access-24qh6\") pod \"node-resolver-4mlb8\" (UID: \"e2c380ad-08a8-4689-b88f-f07b659e6314\") " pod="openshift-dns/node-resolver-4mlb8"
Sep 17 17:57:12.960739 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.960136    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvt5r\" (UniqueName: \"kubernetes.io/projected/f49b2f6e-bffd-4176-9306-863c8f5e7ee3-kube-api-access-nvt5r\") pod \"multus-additional-cni-plugins-kp6mp\" (UID: \"f49b2f6e-bffd-4176-9306-863c8f5e7ee3\") " pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:12.961761 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.961738    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvqwj\" (UniqueName: \"kubernetes.io/projected/3ee045a7-cfc1-40b2-87a8-65bedec89f66-kube-api-access-hvqwj\") pod \"network-metrics-daemon-xjh98\" (UID: \"3ee045a7-cfc1-40b2-87a8-65bedec89f66\") " pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:12.962106 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.962090    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-222v9\" (UniqueName: \"kubernetes.io/projected/f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e-kube-api-access-222v9\") pod \"multus-xfm5k\" (UID: \"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e\") " pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:12.962450 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.962435    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmgjk\" (UniqueName: \"kubernetes.io/projected/53270d0a-dc99-4f59-a2a3-2086917c92f2-kube-api-access-rmgjk\") pod \"ovnkube-node-j2v2z\" (UID: \"53270d0a-dc99-4f59-a2a3-2086917c92f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:12.981781 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:12.981753    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7d3c93d84d7fbd4bfd4bfbc5d80220.slice/crio-1d9f1fb8178150269a28ce098ec650e08fb42bab522df1a9401dc2263d0a1921 WatchSource:0}: Error finding container 1d9f1fb8178150269a28ce098ec650e08fb42bab522df1a9401dc2263d0a1921: Status 404 returned error can't find the container with id 1d9f1fb8178150269a28ce098ec650e08fb42bab522df1a9401dc2263d0a1921
Sep 17 17:57:12.982012 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:12.981997    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58a43d59107368b45168fc33760fb7b.slice/crio-c2b76ed953fdd0970d70c7f28d3884ffe3072a728b005b17fb9f9a631b2a36e2 WatchSource:0}: Error finding container c2b76ed953fdd0970d70c7f28d3884ffe3072a728b005b17fb9f9a631b2a36e2: Status 404 returned error can't find the container with id c2b76ed953fdd0970d70c7f28d3884ffe3072a728b005b17fb9f9a631b2a36e2
Sep 17 17:57:12.987436 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:12.987423    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:57:13.065831 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.065769    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9"
Sep 17 17:57:13.074560 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:13.074538    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b5dcf7_1ca9_4379_b734_2bce67c4d84d.slice/crio-9df4d48c80067fec9752a576824ce1a25a91750e9a92e6f8ec3810ad32688421 WatchSource:0}: Error finding container 9df4d48c80067fec9752a576824ce1a25a91750e9a92e6f8ec3810ad32688421: Status 404 returned error can't find the container with id 9df4d48c80067fec9752a576824ce1a25a91750e9a92e6f8ec3810ad32688421
Sep 17 17:57:13.089024 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.088991    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4mlb8"
Sep 17 17:57:13.095351 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:13.095331    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c380ad_08a8_4689_b88f_f07b659e6314.slice/crio-3521bd9afd1d0d794c1c3d3278155d0b8fb5d14146cbc52e676cdc8487750c02 WatchSource:0}: Error finding container 3521bd9afd1d0d794c1c3d3278155d0b8fb5d14146cbc52e676cdc8487750c02: Status 404 returned error can't find the container with id 3521bd9afd1d0d794c1c3d3278155d0b8fb5d14146cbc52e676cdc8487750c02
Sep 17 17:57:13.103175 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.103160    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vb6r7"
Sep 17 17:57:13.109178 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:13.109160    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030fbe18_1e3d_415f_a113_3e7e1ca71d1b.slice/crio-3706279d570b06504a3499c55b2803d4f464a492a684c64ce8818e848d973ade WatchSource:0}: Error finding container 3706279d570b06504a3499c55b2803d4f464a492a684c64ce8818e848d973ade: Status 404 returned error can't find the container with id 3706279d570b06504a3499c55b2803d4f464a492a684c64ce8818e848d973ade
Sep 17 17:57:13.109732 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.109717    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6jhpj"
Sep 17 17:57:13.116052 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:13.116030    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229dfe1c_397d_48b4_b1d4_e3c5e6429339.slice/crio-a39115e555d14be0be5c2f6100fe077975deb7af18c194eaefcc0c36fdb54886 WatchSource:0}: Error finding container a39115e555d14be0be5c2f6100fe077975deb7af18c194eaefcc0c36fdb54886: Status 404 returned error can't find the container with id a39115e555d14be0be5c2f6100fe077975deb7af18c194eaefcc0c36fdb54886
Sep 17 17:57:13.124443 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.124427    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kp6mp"
Sep 17 17:57:13.131477 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:13.131456    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf49b2f6e_bffd_4176_9306_863c8f5e7ee3.slice/crio-d5339d08f5681d6d2b67a08f228f6563f454a587badeaf529d553f1fd2567183 WatchSource:0}: Error finding container d5339d08f5681d6d2b67a08f228f6563f454a587badeaf529d553f1fd2567183: Status 404 returned error can't find the container with id d5339d08f5681d6d2b67a08f228f6563f454a587badeaf529d553f1fd2567183
Sep 17 17:57:13.142114 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.142098    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mm6bn"
Sep 17 17:57:13.148257 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:13.148236    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc702e515_fb08_4a73_9ac8_3295f2c18596.slice/crio-7568ea3c5d7efd0d5eb54aeb5e13e879fba2039ef10bc9cc325abfa893302323 WatchSource:0}: Error finding container 7568ea3c5d7efd0d5eb54aeb5e13e879fba2039ef10bc9cc325abfa893302323: Status 404 returned error can't find the container with id 7568ea3c5d7efd0d5eb54aeb5e13e879fba2039ef10bc9cc325abfa893302323
Sep 17 17:57:13.156938 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.156922    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:13.162646 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.162630    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4vwlx"
Sep 17 17:57:13.163424 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:13.163400    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53270d0a_dc99_4f59_a2a3_2086917c92f2.slice/crio-765daee3f19507606654c1f896a22e388c363005ddecd48a1a7b4d01624575bf WatchSource:0}: Error finding container 765daee3f19507606654c1f896a22e388c363005ddecd48a1a7b4d01624575bf: Status 404 returned error can't find the container with id 765daee3f19507606654c1f896a22e388c363005ddecd48a1a7b4d01624575bf
Sep 17 17:57:13.167360 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.167341    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xfm5k"
Sep 17 17:57:13.169547 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:13.169530    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fb6002_f122_4cc8_a73b_3d1e45addd6e.slice/crio-249e6725be7b872eaf9a411fa8c5aeb4c0a1ce069fd1c550c15b035dba3cb433 WatchSource:0}: Error finding container 249e6725be7b872eaf9a411fa8c5aeb4c0a1ce069fd1c550c15b035dba3cb433: Status 404 returned error can't find the container with id 249e6725be7b872eaf9a411fa8c5aeb4c0a1ce069fd1c550c15b035dba3cb433
Sep 17 17:57:13.174740 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:13.174724    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c9c2e4_e49b_44f2_ab5c_de1111c9e62e.slice/crio-8be4a370f0ab332d859fbcdf4429fa17ce61101648961e2234dcbc0e32b5848a WatchSource:0}: Error finding container 8be4a370f0ab332d859fbcdf4429fa17ce61101648961e2234dcbc0e32b5848a: Status 404 returned error can't find the container with id 8be4a370f0ab332d859fbcdf4429fa17ce61101648961e2234dcbc0e32b5848a
Sep 17 17:57:13.176144 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:13.176121    2574 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-multus start failed in pod multus-xfm5k_openshift-multus(f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 17 17:57:13.176213 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:13.176158    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-xfm5k" podUID="f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e"
Sep 17 17:57:13.441442 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.441368    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs\") pod \"network-metrics-daemon-xjh98\" (UID: \"3ee045a7-cfc1-40b2-87a8-65bedec89f66\") " pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:13.441613 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.441454    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:13.441613 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:13.441570    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:13.441613 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:13.441563    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:13.441769 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:13.441626    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret podName:c6126893-24bd-4045-a954-7dd893aff7e8 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:14.441611976 +0000 UTC m=+4.742434278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret") pod "global-pull-secret-syncer-4kg4f" (UID: "c6126893-24bd-4045-a954-7dd893aff7e8") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:13.441769 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:13.441646    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs podName:3ee045a7-cfc1-40b2-87a8-65bedec89f66 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:14.441639327 +0000 UTC m=+4.742461628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs") pod "network-metrics-daemon-xjh98" (UID: "3ee045a7-cfc1-40b2-87a8-65bedec89f66") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:13.544583 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.544542    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbctq\" (UniqueName: \"kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq\") pod \"network-check-target-kb4ks\" (UID: \"0081dcb0-3253-477a-8556-da52f047f100\") " pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:13.544746 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:13.544701    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:13.544746 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:13.544720    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:13.544746 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:13.544732    2574 projected.go:196] Error preparing data for projected volume kube-api-access-tbctq for pod openshift-network-diagnostics/network-check-target-kb4ks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:13.544900 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:13.544787    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq podName:0081dcb0-3253-477a-8556-da52f047f100 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:14.544769475 +0000 UTC m=+4.845591802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tbctq" (UniqueName: "kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq") pod "network-check-target-kb4ks" (UID: "0081dcb0-3253-477a-8556-da52f047f100") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:13.598394 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.598366    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:57:13.688766 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.688736    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:57:13.822454 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.822319    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:52:12 +0000 UTC" deadline="2028-02-25 03:12:13.403971428 +0000 UTC"
Sep 17 17:57:13.822454 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.822354    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12609h14m59.581622514s"
Sep 17 17:57:13.953614 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.953556    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp6mp" event={"ID":"f49b2f6e-bffd-4176-9306-863c8f5e7ee3","Type":"ContainerStarted","Data":"d5339d08f5681d6d2b67a08f228f6563f454a587badeaf529d553f1fd2567183"}
Sep 17 17:57:13.964572 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.964498    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4mlb8" event={"ID":"e2c380ad-08a8-4689-b88f-f07b659e6314","Type":"ContainerStarted","Data":"3521bd9afd1d0d794c1c3d3278155d0b8fb5d14146cbc52e676cdc8487750c02"}
Sep 17 17:57:13.966175 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.966146    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal" event={"ID":"f58a43d59107368b45168fc33760fb7b","Type":"ContainerStarted","Data":"c2b76ed953fdd0970d70c7f28d3884ffe3072a728b005b17fb9f9a631b2a36e2"}
Sep 17 17:57:13.990693 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:13.990664    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-0.ec2.internal" event={"ID":"ef7d3c93d84d7fbd4bfd4bfbc5d80220","Type":"ContainerStarted","Data":"1d9f1fb8178150269a28ce098ec650e08fb42bab522df1a9401dc2263d0a1921"}
Sep 17 17:57:14.005054 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.005025    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4vwlx" event={"ID":"a8fb6002-f122-4cc8-a73b-3d1e45addd6e","Type":"ContainerStarted","Data":"249e6725be7b872eaf9a411fa8c5aeb4c0a1ce069fd1c550c15b035dba3cb433"}
Sep 17 17:57:14.007158 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.007128    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z" event={"ID":"53270d0a-dc99-4f59-a2a3-2086917c92f2","Type":"ContainerStarted","Data":"765daee3f19507606654c1f896a22e388c363005ddecd48a1a7b4d01624575bf"}
Sep 17 17:57:14.011646 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.011229    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mm6bn" event={"ID":"c702e515-fb08-4a73-9ac8-3295f2c18596","Type":"ContainerStarted","Data":"7568ea3c5d7efd0d5eb54aeb5e13e879fba2039ef10bc9cc325abfa893302323"}
Sep 17 17:57:14.020313 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.020287    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6jhpj" event={"ID":"229dfe1c-397d-48b4-b1d4-e3c5e6429339","Type":"ContainerStarted","Data":"a39115e555d14be0be5c2f6100fe077975deb7af18c194eaefcc0c36fdb54886"}
Sep 17 17:57:14.024000 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.023974    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vb6r7" event={"ID":"030fbe18-1e3d-415f-a113-3e7e1ca71d1b","Type":"ContainerStarted","Data":"3706279d570b06504a3499c55b2803d4f464a492a684c64ce8818e848d973ade"}
Sep 17 17:57:14.066198 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.066167    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9" event={"ID":"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d","Type":"ContainerStarted","Data":"9df4d48c80067fec9752a576824ce1a25a91750e9a92e6f8ec3810ad32688421"}
Sep 17 17:57:14.087458 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.087391    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xfm5k" event={"ID":"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e","Type":"ContainerStarted","Data":"8be4a370f0ab332d859fbcdf4429fa17ce61101648961e2234dcbc0e32b5848a"}
Sep 17 17:57:14.099966 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:14.099836    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e593011a6c882d153207bb561d02609759cba4aa0f900b0fe4479e5239e479bb\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-xfm5k" podUID="f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e"
Sep 17 17:57:14.452874 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.452763    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:14.452874 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.452818    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs\") pod \"network-metrics-daemon-xjh98\" (UID: \"3ee045a7-cfc1-40b2-87a8-65bedec89f66\") " pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:14.453096 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:14.452965    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:14.453096 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:14.453025    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs podName:3ee045a7-cfc1-40b2-87a8-65bedec89f66 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:16.453007076 +0000 UTC m=+6.753829397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs") pod "network-metrics-daemon-xjh98" (UID: "3ee045a7-cfc1-40b2-87a8-65bedec89f66") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:14.453096 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:14.453089    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:14.453254 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:14.453119    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret podName:c6126893-24bd-4045-a954-7dd893aff7e8 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:16.453108702 +0000 UTC m=+6.753931017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret") pod "global-pull-secret-syncer-4kg4f" (UID: "c6126893-24bd-4045-a954-7dd893aff7e8") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:14.554665 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.553996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbctq\" (UniqueName: \"kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq\") pod \"network-check-target-kb4ks\" (UID: \"0081dcb0-3253-477a-8556-da52f047f100\") " pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:14.554665 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:14.554215    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:14.554665 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:14.554238    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:14.554665 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:14.554251    2574 projected.go:196] Error preparing data for projected volume kube-api-access-tbctq for pod openshift-network-diagnostics/network-check-target-kb4ks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:14.554665 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:14.554308    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq podName:0081dcb0-3253-477a-8556-da52f047f100 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:16.554290068 +0000 UTC m=+6.855112372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tbctq" (UniqueName: "kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq") pod "network-check-target-kb4ks" (UID: "0081dcb0-3253-477a-8556-da52f047f100") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:14.822654 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.822501    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:52:12 +0000 UTC" deadline="2028-02-21 15:54:07.518564373 +0000 UTC"
Sep 17 17:57:14.822654 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.822575    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12525h56m52.695994617s"
Sep 17 17:57:14.932438 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.932403    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:14.932622 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:14.932549    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:14.932681 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.932631    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:14.932751 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:14.932710    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:14.932801 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:14.932793    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:14.932873 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:14.932854    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:15.384715 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:15.384686    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:57:16.470249 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:16.470210    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:16.470718 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:16.470276    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs\") pod \"network-metrics-daemon-xjh98\" (UID: \"3ee045a7-cfc1-40b2-87a8-65bedec89f66\") " pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:16.470718 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:16.470371    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:16.470718 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:16.470460    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret podName:c6126893-24bd-4045-a954-7dd893aff7e8 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:20.470439473 +0000 UTC m=+10.771261778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret") pod "global-pull-secret-syncer-4kg4f" (UID: "c6126893-24bd-4045-a954-7dd893aff7e8") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:16.470718 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:16.470380    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:16.470718 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:16.470544    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs podName:3ee045a7-cfc1-40b2-87a8-65bedec89f66 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:20.470527007 +0000 UTC m=+10.771349328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs") pod "network-metrics-daemon-xjh98" (UID: "3ee045a7-cfc1-40b2-87a8-65bedec89f66") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:16.570646 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:16.570608    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbctq\" (UniqueName: \"kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq\") pod \"network-check-target-kb4ks\" (UID: \"0081dcb0-3253-477a-8556-da52f047f100\") " pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:16.570805 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:16.570783    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:16.570846 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:16.570816    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:16.570846 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:16.570829    2574 projected.go:196] Error preparing data for projected volume kube-api-access-tbctq for pod openshift-network-diagnostics/network-check-target-kb4ks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:16.570903 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:16.570891    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq podName:0081dcb0-3253-477a-8556-da52f047f100 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:20.570873774 +0000 UTC m=+10.871696082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tbctq" (UniqueName: "kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq") pod "network-check-target-kb4ks" (UID: "0081dcb0-3253-477a-8556-da52f047f100") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:16.932846 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:16.932763    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:16.932989 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:16.932957    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:16.933610 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:16.933312    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:16.933610 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:16.933415    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:16.933610 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:16.933459    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:16.933610 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:16.933537    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:18.932217 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:18.932180    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:18.932682 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:18.932318    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:18.932770 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:18.932752    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:18.932855 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:18.932832    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:18.932918 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:18.932897    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:18.932974 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:18.932960    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:20.500647 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:20.500609    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:20.501068 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:20.500662    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs\") pod \"network-metrics-daemon-xjh98\" (UID: \"3ee045a7-cfc1-40b2-87a8-65bedec89f66\") " pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:20.501068 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:20.500744    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:20.501068 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:20.500796    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:20.501068 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:20.500815    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret podName:c6126893-24bd-4045-a954-7dd893aff7e8 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:28.500794512 +0000 UTC m=+18.801616814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret") pod "global-pull-secret-syncer-4kg4f" (UID: "c6126893-24bd-4045-a954-7dd893aff7e8") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:20.501068 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:20.500847    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs podName:3ee045a7-cfc1-40b2-87a8-65bedec89f66 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:28.50083321 +0000 UTC m=+18.801655530 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs") pod "network-metrics-daemon-xjh98" (UID: "3ee045a7-cfc1-40b2-87a8-65bedec89f66") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:20.600959 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:20.600923    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbctq\" (UniqueName: \"kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq\") pod \"network-check-target-kb4ks\" (UID: \"0081dcb0-3253-477a-8556-da52f047f100\") " pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:20.601133 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:20.601105    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:20.601133 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:20.601125    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:20.601293 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:20.601137    2574 projected.go:196] Error preparing data for projected volume kube-api-access-tbctq for pod openshift-network-diagnostics/network-check-target-kb4ks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:20.601293 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:20.601199    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq podName:0081dcb0-3253-477a-8556-da52f047f100 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:28.601180218 +0000 UTC m=+18.902002536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tbctq" (UniqueName: "kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq") pod "network-check-target-kb4ks" (UID: "0081dcb0-3253-477a-8556-da52f047f100") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:20.931962 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:20.931881    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:20.932120 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:20.932010    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:20.932422 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:20.932393    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:20.932543 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:20.932489    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:20.932598 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:20.932547    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:20.932598 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:20.932592    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:22.932573 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:22.932539    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:22.932989 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:22.932559    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:22.932989 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:22.932700    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:22.932989 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:22.932540    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:22.932989 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:22.932760    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:22.932989 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:22.932828    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:24.932324 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:24.932280    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:24.932795 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:24.932280    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:24.932795 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:24.932405    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:24.932795 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:24.932281    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:24.932795 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:24.932548    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:24.932795 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:24.932623    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:26.932708 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:26.932532    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:26.932994 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:26.932551    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:26.932994 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:26.932806    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:26.932994 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:26.932580    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:26.932994 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:26.932849    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:26.933141 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:26.933032    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:27.135945 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:27.135910    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z" event={"ID":"53270d0a-dc99-4f59-a2a3-2086917c92f2","Type":"ContainerStarted","Data":"8100a0b082a0f117cbd0ea43d4e475de018b601eb9dbce65bc14bf65723a1940"}
Sep 17 17:57:27.136086 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:27.135998    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z" event={"ID":"53270d0a-dc99-4f59-a2a3-2086917c92f2","Type":"ContainerStarted","Data":"515bb1d99d8b75319cbc6567d3053ada933901a55c2585fc7c8a5638287660fc"}
Sep 17 17:57:27.138620 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:27.137901    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vb6r7" event={"ID":"030fbe18-1e3d-415f-a113-3e7e1ca71d1b","Type":"ContainerStarted","Data":"28f3c00155308478e53d166a40dd27253bed73aa159afad57a29acb96f7fba1e"}
Sep 17 17:57:27.141981 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:27.141955    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-0.ec2.internal" event={"ID":"ef7d3c93d84d7fbd4bfd4bfbc5d80220","Type":"ContainerStarted","Data":"96be4dd4c61594f9ea471a4ec33c47f1273b82e2cf57e85857b6f9eaebec7f49"}
Sep 17 17:57:27.183193 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:27.183092    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vb6r7" podStartSLOduration=2.613412835 podStartE2EDuration="16.183076228s" totalImagesPullingTime="13.569663393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:27.166094146 +0000 UTC m=+17.466916468" watchObservedRunningTime="2026-09-17 17:57:27.183076228 +0000 UTC m=+17.483898550"
Sep 17 17:57:27.183404 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:27.183377    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-0.ec2.internal" podStartSLOduration=15.183366172 podStartE2EDuration="15.183366172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:27.183239138 +0000 UTC m=+17.484061451" watchObservedRunningTime="2026-09-17 17:57:27.183366172 +0000 UTC m=+17.484188494"
Sep 17 17:57:28.145628 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.145436    2574 generic.go:356] "Generic (PLEG): container finished" podID="f49b2f6e-bffd-4176-9306-863c8f5e7ee3" containerID="5a1b64decf3f2c0c8322884a044e1029e5a4908413a7eebb59e25ed3bb0c17c5" exitCode=0
Sep 17 17:57:28.146325 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.145536    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp6mp" event={"ID":"f49b2f6e-bffd-4176-9306-863c8f5e7ee3","Type":"ContainerDied","Data":"5a1b64decf3f2c0c8322884a044e1029e5a4908413a7eebb59e25ed3bb0c17c5"}
Sep 17 17:57:28.147637 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.147585    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4mlb8" event={"ID":"e2c380ad-08a8-4689-b88f-f07b659e6314","Type":"ContainerStarted","Data":"29048fe0fb0ded13a54e2e51f3fb2a4d4486c0240bc992589e380a2d5786a14a"}
Sep 17 17:57:28.149096 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.149074    2574 generic.go:356] "Generic (PLEG): container finished" podID="f58a43d59107368b45168fc33760fb7b" containerID="6ad7fa191282a62438608c4882584c37c05995b1480648d53ea55060e93fb7b6" exitCode=0
Sep 17 17:57:28.149193 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.149154    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal" event={"ID":"f58a43d59107368b45168fc33760fb7b","Type":"ContainerDied","Data":"6ad7fa191282a62438608c4882584c37c05995b1480648d53ea55060e93fb7b6"}
Sep 17 17:57:28.151175 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.151153    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4vwlx" event={"ID":"a8fb6002-f122-4cc8-a73b-3d1e45addd6e","Type":"ContainerStarted","Data":"9cf5c04099fc19f303582dbf6cc03ef57a81e368e656fdf2be8066185f77096d"}
Sep 17 17:57:28.154475 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.154460    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 17:57:28.154862 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.154841    2574 generic.go:356] "Generic (PLEG): container finished" podID="53270d0a-dc99-4f59-a2a3-2086917c92f2" containerID="8100a0b082a0f117cbd0ea43d4e475de018b601eb9dbce65bc14bf65723a1940" exitCode=1
Sep 17 17:57:28.154951 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.154913    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z" event={"ID":"53270d0a-dc99-4f59-a2a3-2086917c92f2","Type":"ContainerDied","Data":"8100a0b082a0f117cbd0ea43d4e475de018b601eb9dbce65bc14bf65723a1940"}
Sep 17 17:57:28.154997 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.154974    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z" event={"ID":"53270d0a-dc99-4f59-a2a3-2086917c92f2","Type":"ContainerStarted","Data":"969011cb6f7e6dd4ef3497019e794bd93c97c7b02adea30de905c253af86ec56"}
Sep 17 17:57:28.155040 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.155020    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z" event={"ID":"53270d0a-dc99-4f59-a2a3-2086917c92f2","Type":"ContainerStarted","Data":"9fef7a772a042cbe2f15683bb03fa2f577da57b6879b01e75acb0d1a9875de2d"}
Sep 17 17:57:28.155088 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.155057    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z" event={"ID":"53270d0a-dc99-4f59-a2a3-2086917c92f2","Type":"ContainerStarted","Data":"575b034fe94b955438aec58563dda9fca0380775ad8d1d0908a4addadd35d452"}
Sep 17 17:57:28.155135 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.155101    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z" event={"ID":"53270d0a-dc99-4f59-a2a3-2086917c92f2","Type":"ContainerStarted","Data":"5a71b28a07905eb51c53121328dddccaad79ee9fcc41d4e24e9b298f0962cb3e"}
Sep 17 17:57:28.156646 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.156627    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6jhpj" event={"ID":"229dfe1c-397d-48b4-b1d4-e3c5e6429339","Type":"ContainerStarted","Data":"578d9fa8bacb52978c390d6fd8de400bfe8f3439627295108d1f1308967ff252"}
Sep 17 17:57:28.158218 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.158199    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9" event={"ID":"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d","Type":"ContainerStarted","Data":"1ea50aed66249ae9af621e30781351c286b4af3b83278e27ee8c79a48bd4857c"}
Sep 17 17:57:28.218419 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.218372    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6jhpj" podStartSLOduration=2.683827898 podStartE2EDuration="16.218350772s" totalImagesPullingTime="13.534522874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:28.200004698 +0000 UTC m=+18.500827023" watchObservedRunningTime="2026-09-17 17:57:28.218350772 +0000 UTC m=+18.519173096"
Sep 17 17:57:28.219109 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.219071    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4vwlx" podStartSLOduration=2.789334984 podStartE2EDuration="16.219058498s" totalImagesPullingTime="13.429723514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:28.218470639 +0000 UTC m=+18.519292962" watchObservedRunningTime="2026-09-17 17:57:28.219058498 +0000 UTC m=+18.519880822"
Sep 17 17:57:28.234559 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.234502    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4mlb8" podStartSLOduration=3.65068072 podStartE2EDuration="17.234491711s" totalImagesPullingTime="13.583810991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:28.234462626 +0000 UTC m=+18.535284946" watchObservedRunningTime="2026-09-17 17:57:28.234491711 +0000 UTC m=+18.535314035"
Sep 17 17:57:28.551245 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.551162    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:28.551245 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.551225    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs\") pod \"network-metrics-daemon-xjh98\" (UID: \"3ee045a7-cfc1-40b2-87a8-65bedec89f66\") " pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:28.551571 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:28.551352    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:28.551571 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:28.551352    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:28.551571 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:28.551430    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs podName:3ee045a7-cfc1-40b2-87a8-65bedec89f66 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:44.55139927 +0000 UTC m=+34.852221576 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs") pod "network-metrics-daemon-xjh98" (UID: "3ee045a7-cfc1-40b2-87a8-65bedec89f66") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:28.551571 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:28.551456    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret podName:c6126893-24bd-4045-a954-7dd893aff7e8 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:44.551436715 +0000 UTC m=+34.852259043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret") pod "global-pull-secret-syncer-4kg4f" (UID: "c6126893-24bd-4045-a954-7dd893aff7e8") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:28.652622 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.652581    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbctq\" (UniqueName: \"kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq\") pod \"network-check-target-kb4ks\" (UID: \"0081dcb0-3253-477a-8556-da52f047f100\") " pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:28.652794 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:28.652726    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:28.652794 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:28.652748    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:28.652794 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:28.652760    2574 projected.go:196] Error preparing data for projected volume kube-api-access-tbctq for pod openshift-network-diagnostics/network-check-target-kb4ks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:28.652949 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:28.652830    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq podName:0081dcb0-3253-477a-8556-da52f047f100 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:44.652811397 +0000 UTC m=+34.953633715 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tbctq" (UniqueName: "kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq") pod "network-check-target-kb4ks" (UID: "0081dcb0-3253-477a-8556-da52f047f100") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:28.809462 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.809171    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:28.900794 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.900754    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:28.933128 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.932838    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:28.933128 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:28.932926    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:28.933128 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.932986    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:28.933128 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:28.933024    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:28.933128 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:28.933067    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:28.933128 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:28.933104    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:29.002674 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:29.002646    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:29.057744 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:29.057725    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:57:29.162652 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:29.162546    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9" event={"ID":"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d","Type":"ContainerStarted","Data":"c61beaa70bdf20a01f92aefcc700f906d4d5b61733046cc1d88ee0f20d6581bd"}
Sep 17 17:57:29.165442 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:29.165013    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal" event={"ID":"f58a43d59107368b45168fc33760fb7b","Type":"ContainerStarted","Data":"5eca8464aae07b3b744cd01869cd27d6288fb1a1d06069cd9fd592cdab2369b3"}
Sep 17 17:57:29.167123 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:29.167099    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mm6bn" event={"ID":"c702e515-fb08-4a73-9ac8-3295f2c18596","Type":"ContainerStarted","Data":"a32e461098eba8e3be1726ac6f686b5fee351bff226b68ab8110f38db42e11ad"}
Sep 17 17:57:29.201263 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:29.201213    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-0.ec2.internal" podStartSLOduration=17.201195167 podStartE2EDuration="17.201195167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:29.18259724 +0000 UTC m=+19.483419567" watchObservedRunningTime="2026-09-17 17:57:29.201195167 +0000 UTC m=+19.502017487"
Sep 17 17:57:29.334730 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:29.334700    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:29.436130 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:29.436047    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:29.812586 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:29.812468    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:57:29.057743043Z","UUID":"6fc1acfd-f2af-491b-87e2-bd2f2819e38c","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:57:29.814922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:29.814556    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:57:29.814922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:29.814589    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:57:30.173444 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:30.173369    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 17:57:30.173876 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:30.173855    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z" event={"ID":"53270d0a-dc99-4f59-a2a3-2086917c92f2","Type":"ContainerStarted","Data":"ed89104af6385c73fd990ae21306f5cd3b2406004518d394b855a53098ecae62"}
Sep 17 17:57:30.176898 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:30.176439    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9" event={"ID":"e6b5dcf7-1ca9-4379-b734-2bce67c4d84d","Type":"ContainerStarted","Data":"8a06b34726332e0cb1f82a09de011ed9694af295bb69c608e0b294286b20c274"}
Sep 17 17:57:30.198107 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:30.198065    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dc2f9" podStartSLOduration=3.86219678 podStartE2EDuration="19.19804645s" totalImagesPullingTime="15.33584967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:30.197147442 +0000 UTC m=+20.497969946" watchObservedRunningTime="2026-09-17 17:57:30.19804645 +0000 UTC m=+20.498868776"
Sep 17 17:57:30.198434 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:30.198399    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mm6bn" podStartSLOduration=4.670829727 podStartE2EDuration="18.198389493s" totalImagesPullingTime="13.527559766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:29.201422769 +0000 UTC m=+19.502245106" watchObservedRunningTime="2026-09-17 17:57:30.198389493 +0000 UTC m=+20.499211816"
Sep 17 17:57:30.496621 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:30.496548    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6jhpj"
Sep 17 17:57:30.497303 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:30.497280    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6jhpj"
Sep 17 17:57:30.932572 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:30.932486    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:30.932696 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:30.932624    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:30.932696 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:30.932643    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:30.932801 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:30.932756    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:30.932856 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:30.932806    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:30.932856 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:30.932848    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:30.954901 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:30.954877    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:31.061175 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:31.061142    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:32.932408 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:32.932380    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:32.932408 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:32.932408    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:32.932889 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:32.932415    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:32.932889 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:32.932503    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:32.932889 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:32.932582    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:32.932889 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:32.932690    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:34.932588 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:34.932555    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:34.933015 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:34.932553    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:34.933015 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:34.932678    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:34.933015 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:34.932748    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:34.933015 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:34.932556    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:34.933015 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:34.932835    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:36.932382 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:36.932193    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:36.932967 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:36.932209    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:36.932967 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:36.932458    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:36.932967 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:36.932213    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:36.932967 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:36.932552    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:36.932967 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:36.932626    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:37.193676 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:37.193595    2574 generic.go:356] "Generic (PLEG): container finished" podID="f49b2f6e-bffd-4176-9306-863c8f5e7ee3" containerID="4b5bc1c2ecaa4fcfeb9697042e8a618d8bc59787e1240e82d7769824e97cd29c" exitCode=0
Sep 17 17:57:37.193676 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:37.193656    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp6mp" event={"ID":"f49b2f6e-bffd-4176-9306-863c8f5e7ee3","Type":"ContainerDied","Data":"4b5bc1c2ecaa4fcfeb9697042e8a618d8bc59787e1240e82d7769824e97cd29c"}
Sep 17 17:57:37.197159 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:37.197145    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 17:57:37.197550 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:37.197508    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z" event={"ID":"53270d0a-dc99-4f59-a2a3-2086917c92f2","Type":"ContainerStarted","Data":"e7a26324eeeb873d4115d4512f4856df15bb20b17ce64dbe264d7e02aba4c159"}
Sep 17 17:57:37.197748 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:37.197730    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:37.197882 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:37.197863    2574 scope.go:117] "RemoveContainer" containerID="8100a0b082a0f117cbd0ea43d4e475de018b601eb9dbce65bc14bf65723a1940"
Sep 17 17:57:37.199087 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:37.199071    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xfm5k" event={"ID":"f3c9c2e4-e49b-44f2-ab5c-de1111c9e62e","Type":"ContainerStarted","Data":"d578f4a65a71837d754bd311d0eb5622ad0c409fcaa09d4783f8af65ce1c60f7"}
Sep 17 17:57:37.213401 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:37.213378    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:37.332538 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:37.330945    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xfm5k" podStartSLOduration=1.904444365 podStartE2EDuration="25.330924687s" totalImagesPullingTime="23.426480322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-17 17:57:37.272357356 +0000 UTC m=+27.573179705" watchObservedRunningTime="2026-09-17 17:57:37.330924687 +0000 UTC m=+27.631747010"
Sep 17 17:57:37.511729 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:37.511637    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:37.512691 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:37.512657    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:37.683820 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:37.683794    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:38.204976 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.204911    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 17:57:38.205416 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.205250    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z" event={"ID":"53270d0a-dc99-4f59-a2a3-2086917c92f2","Type":"ContainerStarted","Data":"ca093ee219a62d2f318d165b7bc4421e63fe38819bfffa96973c3a6345c4e8fb"}
Sep 17 17:57:38.205416 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.205407    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:38.207416 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.207381    2574 generic.go:356] "Generic (PLEG): container finished" podID="f49b2f6e-bffd-4176-9306-863c8f5e7ee3" containerID="ac66ffe8099b4a707a1fc60bd31512a3ad0bce46d80c35c44594573c9cc086ab" exitCode=0
Sep 17 17:57:38.207533 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.207423    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp6mp" event={"ID":"f49b2f6e-bffd-4176-9306-863c8f5e7ee3","Type":"ContainerDied","Data":"ac66ffe8099b4a707a1fc60bd31512a3ad0bce46d80c35c44594573c9cc086ab"}
Sep 17 17:57:38.221254 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.221218    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:57:38.237413 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.237387    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6jhpj"
Sep 17 17:57:38.237552 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.237497    2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:57:38.238005 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.237981    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6jhpj"
Sep 17 17:57:38.241965 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.241919    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z" podStartSLOduration=12.637670041 podStartE2EDuration="26.241901572s" totalImagesPullingTime="13.604231531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:38.241265013 +0000 UTC m=+28.542087372" watchObservedRunningTime="2026-09-17 17:57:38.241901572 +0000 UTC m=+28.542723896"
Sep 17 17:57:38.474110 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.474036    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:38.644327 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.644080    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:38.742796 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.742670    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kb4ks"]
Sep 17 17:57:38.742951 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.742806    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:38.742951 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:38.742916    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:38.745297 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.745272    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4kg4f"]
Sep 17 17:57:38.745439 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.745363    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:38.745439 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:38.745426    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:38.750601 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.750545    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xjh98"]
Sep 17 17:57:38.750796 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.750722    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:38.750861 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:38.750819    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:38.868375 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:38.868348    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:39.213130 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:39.213102    2574 generic.go:356] "Generic (PLEG): container finished" podID="f49b2f6e-bffd-4176-9306-863c8f5e7ee3" containerID="4ca8c080922d0c4bd00e093dc64a7b5748f032f2e0eee0f52878d73a8ff5cb77" exitCode=0
Sep 17 17:57:39.213643 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:39.213184    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp6mp" event={"ID":"f49b2f6e-bffd-4176-9306-863c8f5e7ee3","Type":"ContainerDied","Data":"4ca8c080922d0c4bd00e093dc64a7b5748f032f2e0eee0f52878d73a8ff5cb77"}
Sep 17 17:57:40.932459 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:40.932419    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:40.932816 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:40.932603    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xjh98" podUID="3ee045a7-cfc1-40b2-87a8-65bedec89f66"
Sep 17 17:57:40.932816 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:40.932422    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:40.932816 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:40.932717    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4kg4f" podUID="c6126893-24bd-4045-a954-7dd893aff7e8"
Sep 17 17:57:40.932816 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:40.932422    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:40.932816 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:40.932791    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kb4ks" podUID="0081dcb0-3253-477a-8556-da52f047f100"
Sep 17 17:57:42.255136 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.255102    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-0.ec2.internal" event="NodeReady"
Sep 17 17:57:42.255588 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.255247    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:57:42.357224 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.357145    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pbkvq"]
Sep 17 17:57:42.397281 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.397247    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pbkvq"]
Sep 17 17:57:42.397451 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.397373    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:42.401365 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.401339    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4ffrq\""
Sep 17 17:57:42.401497 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.401354    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:57:42.411444 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.411424    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:57:42.450948 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.450910    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jl2b8"]
Sep 17 17:57:42.456380 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.456356    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4rfr\" (UniqueName: \"kubernetes.io/projected/da415339-7f86-4698-94a7-625f0e5015b2-kube-api-access-s4rfr\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:42.456538 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.456413    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:42.456538 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.456435    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da415339-7f86-4698-94a7-625f0e5015b2-config-volume\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:42.456538 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.456499    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da415339-7f86-4698-94a7-625f0e5015b2-tmp-dir\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:42.470827 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.470802    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jl2b8"
Sep 17 17:57:42.472866 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.472844    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jl2b8"]
Sep 17 17:57:42.474740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.474390    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:57:42.474740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.474420    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mkswj\""
Sep 17 17:57:42.474740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.474696    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:57:42.474740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.474700    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:57:42.557870 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.557839    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4rfr\" (UniqueName: \"kubernetes.io/projected/da415339-7f86-4698-94a7-625f0e5015b2-kube-api-access-s4rfr\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:42.558032 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.557904    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf9e50fc-d402-4599-8278-cc186d5a07ab-cert\") pod \"ingress-canary-jl2b8\" (UID: \"bf9e50fc-d402-4599-8278-cc186d5a07ab\") " pod="openshift-ingress-canary/ingress-canary-jl2b8"
Sep 17 17:57:42.558032 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.557941    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:42.558032 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.557960    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dbfp\" (UniqueName: \"kubernetes.io/projected/bf9e50fc-d402-4599-8278-cc186d5a07ab-kube-api-access-2dbfp\") pod \"ingress-canary-jl2b8\" (UID: \"bf9e50fc-d402-4599-8278-cc186d5a07ab\") " pod="openshift-ingress-canary/ingress-canary-jl2b8"
Sep 17 17:57:42.558032 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.557990    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da415339-7f86-4698-94a7-625f0e5015b2-config-volume\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:42.558032 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.558024    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da415339-7f86-4698-94a7-625f0e5015b2-tmp-dir\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:42.558282 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:42.558116    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:42.558282 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:42.558177    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls podName:da415339-7f86-4698-94a7-625f0e5015b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:43.058159883 +0000 UTC m=+33.358982184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls") pod "dns-default-pbkvq" (UID: "da415339-7f86-4698-94a7-625f0e5015b2") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:42.558361 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.558347    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da415339-7f86-4698-94a7-625f0e5015b2-tmp-dir\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:42.558616 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.558589    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da415339-7f86-4698-94a7-625f0e5015b2-config-volume\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:42.574320 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.574294    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4rfr\" (UniqueName: \"kubernetes.io/projected/da415339-7f86-4698-94a7-625f0e5015b2-kube-api-access-s4rfr\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:42.659088 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.659008    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf9e50fc-d402-4599-8278-cc186d5a07ab-cert\") pod \"ingress-canary-jl2b8\" (UID: \"bf9e50fc-d402-4599-8278-cc186d5a07ab\") " pod="openshift-ingress-canary/ingress-canary-jl2b8"
Sep 17 17:57:42.659088 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.659074    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dbfp\" (UniqueName: \"kubernetes.io/projected/bf9e50fc-d402-4599-8278-cc186d5a07ab-kube-api-access-2dbfp\") pod \"ingress-canary-jl2b8\" (UID: \"bf9e50fc-d402-4599-8278-cc186d5a07ab\") " pod="openshift-ingress-canary/ingress-canary-jl2b8"
Sep 17 17:57:42.661885 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.661857    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf9e50fc-d402-4599-8278-cc186d5a07ab-cert\") pod \"ingress-canary-jl2b8\" (UID: \"bf9e50fc-d402-4599-8278-cc186d5a07ab\") " pod="openshift-ingress-canary/ingress-canary-jl2b8"
Sep 17 17:57:42.672670 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.672646    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dbfp\" (UniqueName: \"kubernetes.io/projected/bf9e50fc-d402-4599-8278-cc186d5a07ab-kube-api-access-2dbfp\") pod \"ingress-canary-jl2b8\" (UID: \"bf9e50fc-d402-4599-8278-cc186d5a07ab\") " pod="openshift-ingress-canary/ingress-canary-jl2b8"
Sep 17 17:57:42.712872 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.712831    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:42.781863 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.781832    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jl2b8"
Sep 17 17:57:42.851503 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.851471    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:42.932915 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.932681    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:42.933070 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.932688    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:42.933070 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.932754    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:42.936587 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.936560    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:57:42.936742 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.936640    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:57:42.936742 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.936647    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:57:42.936742 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.936696    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:57:42.936899 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.936833    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m4z79\""
Sep 17 17:57:42.937404 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:42.937385    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cf4sb\""
Sep 17 17:57:43.062993 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:43.062951    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:43.063166 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:43.063130    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:43.063234 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:43.063205    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls podName:da415339-7f86-4698-94a7-625f0e5015b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:44.063187381 +0000 UTC m=+34.364009686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls") pod "dns-default-pbkvq" (UID: "da415339-7f86-4698-94a7-625f0e5015b2") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:43.585601 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:43.585576    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jl2b8"]
Sep 17 17:57:43.589373 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:43.589349    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9e50fc_d402_4599_8278_cc186d5a07ab.slice/crio-64c7896bc24fde220e5adc3767419b1d50835a02b27a9ceaad033b3d2348c77f WatchSource:0}: Error finding container 64c7896bc24fde220e5adc3767419b1d50835a02b27a9ceaad033b3d2348c77f: Status 404 returned error can't find the container with id 64c7896bc24fde220e5adc3767419b1d50835a02b27a9ceaad033b3d2348c77f
Sep 17 17:57:44.074095 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:44.074044    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:44.074260 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:44.074200    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:44.074309 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:44.074274    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls podName:da415339-7f86-4698-94a7-625f0e5015b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:46.074256559 +0000 UTC m=+36.375078874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls") pod "dns-default-pbkvq" (UID: "da415339-7f86-4698-94a7-625f0e5015b2") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:44.232065 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:44.231711    2574 generic.go:356] "Generic (PLEG): container finished" podID="f49b2f6e-bffd-4176-9306-863c8f5e7ee3" containerID="7408617205c32553203e75bcca4564b4f0f8b3b0421a800a887cb6f3619808bd" exitCode=0
Sep 17 17:57:44.232065 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:44.231813    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp6mp" event={"ID":"f49b2f6e-bffd-4176-9306-863c8f5e7ee3","Type":"ContainerDied","Data":"7408617205c32553203e75bcca4564b4f0f8b3b0421a800a887cb6f3619808bd"}
Sep 17 17:57:44.233283 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:44.233254    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jl2b8" event={"ID":"bf9e50fc-d402-4599-8278-cc186d5a07ab","Type":"ContainerStarted","Data":"64c7896bc24fde220e5adc3767419b1d50835a02b27a9ceaad033b3d2348c77f"}
Sep 17 17:57:44.578922 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:44.578891    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:44.579107 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:44.578934    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs\") pod \"network-metrics-daemon-xjh98\" (UID: \"3ee045a7-cfc1-40b2-87a8-65bedec89f66\") " pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:57:44.579107 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:44.579066    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:57:44.579213 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:44.579129    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs podName:3ee045a7-cfc1-40b2-87a8-65bedec89f66 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:16.579109159 +0000 UTC m=+66.879931462 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs") pod "network-metrics-daemon-xjh98" (UID: "3ee045a7-cfc1-40b2-87a8-65bedec89f66") : secret "metrics-daemon-secret" not found
Sep 17 17:57:44.582738 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:44.582713    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6126893-24bd-4045-a954-7dd893aff7e8-original-pull-secret\") pod \"global-pull-secret-syncer-4kg4f\" (UID: \"c6126893-24bd-4045-a954-7dd893aff7e8\") " pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:44.679745 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:44.679657    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbctq\" (UniqueName: \"kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq\") pod \"network-check-target-kb4ks\" (UID: \"0081dcb0-3253-477a-8556-da52f047f100\") " pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:44.682580 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:44.682549    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbctq\" (UniqueName: \"kubernetes.io/projected/0081dcb0-3253-477a-8556-da52f047f100-kube-api-access-tbctq\") pod \"network-check-target-kb4ks\" (UID: \"0081dcb0-3253-477a-8556-da52f047f100\") " pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:44.752791 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:44.752756    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:44.758945 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:44.758920    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4kg4f"
Sep 17 17:57:45.113136 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:45.113096    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4kg4f"]
Sep 17 17:57:45.121816 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:45.121789    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kb4ks"]
Sep 17 17:57:45.239764 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:45.239726    2574 generic.go:356] "Generic (PLEG): container finished" podID="f49b2f6e-bffd-4176-9306-863c8f5e7ee3" containerID="4293c67e33ef6cc04c405262aa1b9e9b12d3b582ced697f7aa180e3e8aeb18c1" exitCode=0
Sep 17 17:57:45.239921 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:45.239774    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp6mp" event={"ID":"f49b2f6e-bffd-4176-9306-863c8f5e7ee3","Type":"ContainerDied","Data":"4293c67e33ef6cc04c405262aa1b9e9b12d3b582ced697f7aa180e3e8aeb18c1"}
Sep 17 17:57:45.759532 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:45.759476    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6126893_24bd_4045_a954_7dd893aff7e8.slice/crio-273a91a83707225aed3814ab4431f63947a53fe2227a2ee9e4a1ebebf443bc47 WatchSource:0}: Error finding container 273a91a83707225aed3814ab4431f63947a53fe2227a2ee9e4a1ebebf443bc47: Status 404 returned error can't find the container with id 273a91a83707225aed3814ab4431f63947a53fe2227a2ee9e4a1ebebf443bc47
Sep 17 17:57:45.760459 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:57:45.760055    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0081dcb0_3253_477a_8556_da52f047f100.slice/crio-e22c54c0b00c54f1c8cc551eda71ed5eb5d0e80d0e04a1273e4ff384ec7d5cfe WatchSource:0}: Error finding container e22c54c0b00c54f1c8cc551eda71ed5eb5d0e80d0e04a1273e4ff384ec7d5cfe: Status 404 returned error can't find the container with id e22c54c0b00c54f1c8cc551eda71ed5eb5d0e80d0e04a1273e4ff384ec7d5cfe
Sep 17 17:57:46.090473 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:46.090445    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:46.090584 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:46.090571    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:46.090647 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:46.090621    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls podName:da415339-7f86-4698-94a7-625f0e5015b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:50.090608612 +0000 UTC m=+40.391430913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls") pod "dns-default-pbkvq" (UID: "da415339-7f86-4698-94a7-625f0e5015b2") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:46.193537 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:46.193487    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:46.244288 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:46.244247    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4kg4f" event={"ID":"c6126893-24bd-4045-a954-7dd893aff7e8","Type":"ContainerStarted","Data":"273a91a83707225aed3814ab4431f63947a53fe2227a2ee9e4a1ebebf443bc47"}
Sep 17 17:57:46.248424 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:46.248369    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp6mp" event={"ID":"f49b2f6e-bffd-4176-9306-863c8f5e7ee3","Type":"ContainerStarted","Data":"aca5d94c6b385e00d4988e346bb47cf600bb88840b68aeac09a745f0e8ff5cd3"}
Sep 17 17:57:46.250528 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:46.250491    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jl2b8" event={"ID":"bf9e50fc-d402-4599-8278-cc186d5a07ab","Type":"ContainerStarted","Data":"ccd7c14f8a77e3e92b25101450fc3e1f5bf80de1a1afcfa12552a1f7c2856013"}
Sep 17 17:57:46.251976 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:46.251955    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kb4ks" event={"ID":"0081dcb0-3253-477a-8556-da52f047f100","Type":"ContainerStarted","Data":"e22c54c0b00c54f1c8cc551eda71ed5eb5d0e80d0e04a1273e4ff384ec7d5cfe"}
Sep 17 17:57:46.277343 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:46.277286    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kp6mp" podStartSLOduration=6.6901515 podStartE2EDuration="34.277265751s" totalImagesPullingTime="27.587114251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:46.276632712 +0000 UTC m=+36.577455061" watchObservedRunningTime="2026-09-17 17:57:46.277265751 +0000 UTC m=+36.578088076"
Sep 17 17:57:46.282205 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:46.282137    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:46.296840 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:46.296756    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jl2b8" podStartSLOduration=1.842426445 podStartE2EDuration="4.296739409s" totalImagesPullingTime="2.454312964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:46.295643548 +0000 UTC m=+36.596465884" watchObservedRunningTime="2026-09-17 17:57:46.296739409 +0000 UTC m=+36.597561733"
Sep 17 17:57:46.409402 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:46.409367    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:46.491536 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:46.491490    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:46.585838 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:46.585802    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:50.119500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:50.119455    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:50.119942 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:50.119613    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:50.119942 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:50.119686    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls podName:da415339-7f86-4698-94a7-625f0e5015b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:58.119665931 +0000 UTC m=+48.420488254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls") pod "dns-default-pbkvq" (UID: "da415339-7f86-4698-94a7-625f0e5015b2") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:51.266010 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:51.265979    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4kg4f" event={"ID":"c6126893-24bd-4045-a954-7dd893aff7e8","Type":"ContainerStarted","Data":"a27b04efd27010bf89c3cf1c2ce4efdc02dc45e28adba3d8d92e79973cc0e17e"}
Sep 17 17:57:51.267412 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:51.267388    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kb4ks" event={"ID":"0081dcb0-3253-477a-8556-da52f047f100","Type":"ContainerStarted","Data":"ba52ffa88d25b2c7f8472a1b01030029856204c32286c799451da34f220f4fcf"}
Sep 17 17:57:51.267552 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:51.267537    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:57:51.287038 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:51.286987    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4kg4f" podStartSLOduration=34.681586725 podStartE2EDuration="39.286975192s" totalImagesPullingTime="4.605388467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:51.285938677 +0000 UTC m=+41.586761001" watchObservedRunningTime="2026-09-17 17:57:51.286975192 +0000 UTC m=+41.587797505"
Sep 17 17:57:51.843251 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:51.843222    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:51.950020 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:51.949994    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:58.170176 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:57:58.170142    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:57:58.170603 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:58.170284    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:58.170603 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:57:58.170339    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls podName:da415339-7f86-4698-94a7-625f0e5015b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:14.170325942 +0000 UTC m=+64.471148244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls") pod "dns-default-pbkvq" (UID: "da415339-7f86-4698-94a7-625f0e5015b2") : secret "dns-default-metrics-tls" not found
Sep 17 17:58:00.707959 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.707900    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kb4ks" podStartSLOduration=44.11299178 podStartE2EDuration="48.707883665s" totalImagesPullingTime="4.594891885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:51.305639393 +0000 UTC m=+41.606461715" watchObservedRunningTime="2026-09-17 17:58:00.707883665 +0000 UTC m=+51.008705987"
Sep 17 17:58:00.708400 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.708048    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-flfs2"]
Sep 17 17:58:00.710079 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.710060    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.714388 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.714360    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:58:00.714505 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.714442    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 17:58:00.715340 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.715321    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pr782\""
Sep 17 17:58:00.717909 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.716807    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 17:58:00.720738 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.718677    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:58:00.723244 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.723224    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 17:58:00.738679 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.738654    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-flfs2"]
Sep 17 17:58:00.746316 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.746292    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6"]
Sep 17 17:58:00.748118 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.748105    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6"
Sep 17 17:58:00.750138 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.750117    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-tqbd2"]
Sep 17 17:58:00.750758 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.750741    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 17:58:00.751039 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.751021    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 17:58:00.751125 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.751088    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:58:00.751286 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.751266    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 17:58:00.751384 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.751340    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pdwjn\""
Sep 17 17:58:00.751925 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.751910    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-tqbd2"
Sep 17 17:58:00.755062 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.754916    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7wkj2\""
Sep 17 17:58:00.755062 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.754991    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 17:58:00.755062 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.755050    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:58:00.768962 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.768941    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-tqbd2"]
Sep 17 17:58:00.769685 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.769653    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6"]
Sep 17 17:58:00.788755 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.788730    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04d65cc5-2f68-45e9-8359-fff61bd12996-tmp\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.788755 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.788759    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04d65cc5-2f68-45e9-8359-fff61bd12996-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.788936 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.788775    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04d65cc5-2f68-45e9-8359-fff61bd12996-serving-cert\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.788936 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.788792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjt8b\" (UniqueName: \"kubernetes.io/projected/04d65cc5-2f68-45e9-8359-fff61bd12996-kube-api-access-pjt8b\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.788936 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.788809    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04d65cc5-2f68-45e9-8359-fff61bd12996-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.788936 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.788927    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/04d65cc5-2f68-45e9-8359-fff61bd12996-snapshots\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.818700 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.818664    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-l26bh"]
Sep 17 17:58:00.820745 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.820728    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65d8456986-md2f4"]
Sep 17 17:58:00.820865 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.820850    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-l26bh"
Sep 17 17:58:00.822580 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.822558    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:00.823139 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.823124    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-b86wd\""
Sep 17 17:58:00.826015 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.825991    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-skznp\""
Sep 17 17:58:00.826275 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.826254    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:58:00.826435 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.826419    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:58:00.826563 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.826448    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:58:00.832877 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.832858    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:58:00.846626 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.846604    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-l26bh"]
Sep 17 17:58:00.847616 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.847596    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65d8456986-md2f4"]
Sep 17 17:58:00.889291 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.889255    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04d65cc5-2f68-45e9-8359-fff61bd12996-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.889291 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.889298    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm9z9\" (UniqueName: \"kubernetes.io/projected/8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5-kube-api-access-hm9z9\") pod \"kube-storage-version-migrator-operator-55889f7b47-4bbj6\" (UID: \"8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6"
Sep 17 17:58:00.889570 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.889340    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-4bbj6\" (UID: \"8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6"
Sep 17 17:58:00.889570 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.889356    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-4bbj6\" (UID: \"8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6"
Sep 17 17:58:00.889570 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.889378    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/04d65cc5-2f68-45e9-8359-fff61bd12996-snapshots\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.889570 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.889463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04d65cc5-2f68-45e9-8359-fff61bd12996-tmp\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.889570 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.889495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04d65cc5-2f68-45e9-8359-fff61bd12996-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.889570 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.889540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04d65cc5-2f68-45e9-8359-fff61bd12996-serving-cert\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.889570 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.889570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjt8b\" (UniqueName: \"kubernetes.io/projected/04d65cc5-2f68-45e9-8359-fff61bd12996-kube-api-access-pjt8b\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.889893 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.889598    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b22vx\" (UniqueName: \"kubernetes.io/projected/39c9f091-33e4-4108-beec-d30e38305a30-kube-api-access-b22vx\") pod \"volume-data-source-validator-5c4db8bdf9-tqbd2\" (UID: \"39c9f091-33e4-4108-beec-d30e38305a30\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-tqbd2"
Sep 17 17:58:00.889989 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.889971    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04d65cc5-2f68-45e9-8359-fff61bd12996-tmp\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.890067 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.890052    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04d65cc5-2f68-45e9-8359-fff61bd12996-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.890103 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.890073    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/04d65cc5-2f68-45e9-8359-fff61bd12996-snapshots\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.890338 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.890319    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04d65cc5-2f68-45e9-8359-fff61bd12996-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.893021 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.893000    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04d65cc5-2f68-45e9-8359-fff61bd12996-serving-cert\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.907769 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.904693    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjt8b\" (UniqueName: \"kubernetes.io/projected/04d65cc5-2f68-45e9-8359-fff61bd12996-kube-api-access-pjt8b\") pod \"insights-operator-6f9c644dd9-flfs2\" (UID: \"04d65cc5-2f68-45e9-8359-fff61bd12996\") " pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:00.990740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.990646    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab82597c-dd5c-45ee-86c7-51768a677041-registry-certificates\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:00.990740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.990687    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab82597c-dd5c-45ee-86c7-51768a677041-trusted-ca\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:00.990740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.990704    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab82597c-dd5c-45ee-86c7-51768a677041-installation-pull-secrets\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:00.990740 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.990724    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2zs6\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-kube-api-access-x2zs6\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:00.991061 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.990803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b22vx\" (UniqueName: \"kubernetes.io/projected/39c9f091-33e4-4108-beec-d30e38305a30-kube-api-access-b22vx\") pod \"volume-data-source-validator-5c4db8bdf9-tqbd2\" (UID: \"39c9f091-33e4-4108-beec-d30e38305a30\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-tqbd2"
Sep 17 17:58:00.991061 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.990852    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm9z9\" (UniqueName: \"kubernetes.io/projected/8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5-kube-api-access-hm9z9\") pod \"kube-storage-version-migrator-operator-55889f7b47-4bbj6\" (UID: \"8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6"
Sep 17 17:58:00.991061 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.990883    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab82597c-dd5c-45ee-86c7-51768a677041-ca-trust-extracted\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:00.991061 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.990911    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-bound-sa-token\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:00.991061 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.990972    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w5b2\" (UniqueName: \"kubernetes.io/projected/bdc1b1df-104d-4e22-9733-71a65b769d25-kube-api-access-8w5b2\") pod \"network-check-source-7cbc45fddc-l26bh\" (UID: \"bdc1b1df-104d-4e22-9733-71a65b769d25\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-l26bh"
Sep 17 17:58:00.991061 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.991005    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-4bbj6\" (UID: \"8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6"
Sep 17 17:58:00.991061 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.991040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-4bbj6\" (UID: \"8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6"
Sep 17 17:58:00.991061 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.991071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab82597c-dd5c-45ee-86c7-51768a677041-image-registry-private-configuration\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:00.991349 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.991095    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:00.991560 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.991541    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-4bbj6\" (UID: \"8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6"
Sep 17 17:58:00.993301 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:00.993286    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-4bbj6\" (UID: \"8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6"
Sep 17 17:58:01.006973 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.006932    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm9z9\" (UniqueName: \"kubernetes.io/projected/8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5-kube-api-access-hm9z9\") pod \"kube-storage-version-migrator-operator-55889f7b47-4bbj6\" (UID: \"8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6"
Sep 17 17:58:01.007085 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.007012    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b22vx\" (UniqueName: \"kubernetes.io/projected/39c9f091-33e4-4108-beec-d30e38305a30-kube-api-access-b22vx\") pod \"volume-data-source-validator-5c4db8bdf9-tqbd2\" (UID: \"39c9f091-33e4-4108-beec-d30e38305a30\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-tqbd2"
Sep 17 17:58:01.020949 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.020926    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-flfs2"
Sep 17 17:58:01.057614 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.057577    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6"
Sep 17 17:58:01.062827 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.062797    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-tqbd2"
Sep 17 17:58:01.091954 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.091923    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab82597c-dd5c-45ee-86c7-51768a677041-ca-trust-extracted\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.092086 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.091961    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-bound-sa-token\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.092086 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.091983    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w5b2\" (UniqueName: \"kubernetes.io/projected/bdc1b1df-104d-4e22-9733-71a65b769d25-kube-api-access-8w5b2\") pod \"network-check-source-7cbc45fddc-l26bh\" (UID: \"bdc1b1df-104d-4e22-9733-71a65b769d25\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-l26bh"
Sep 17 17:58:01.092086 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.092005    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab82597c-dd5c-45ee-86c7-51768a677041-image-registry-private-configuration\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.092086 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.092021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.092086 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.092062    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab82597c-dd5c-45ee-86c7-51768a677041-registry-certificates\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.092086 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.092076    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab82597c-dd5c-45ee-86c7-51768a677041-trusted-ca\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.092086 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.092089    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab82597c-dd5c-45ee-86c7-51768a677041-installation-pull-secrets\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.092703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.092109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2zs6\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-kube-api-access-x2zs6\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.092703 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.092629    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab82597c-dd5c-45ee-86c7-51768a677041-ca-trust-extracted\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.092799 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:01.092702    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:58:01.092799 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:01.092724    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d8456986-md2f4: secret "image-registry-tls" not found
Sep 17 17:58:01.092896 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:01.092806    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls podName:ab82597c-dd5c-45ee-86c7-51768a677041 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:01.592784365 +0000 UTC m=+51.893606692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls") pod "image-registry-65d8456986-md2f4" (UID: "ab82597c-dd5c-45ee-86c7-51768a677041") : secret "image-registry-tls" not found
Sep 17 17:58:01.093011 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.092990    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab82597c-dd5c-45ee-86c7-51768a677041-registry-certificates\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.093943 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.093904    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab82597c-dd5c-45ee-86c7-51768a677041-trusted-ca\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.094894 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.094877    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab82597c-dd5c-45ee-86c7-51768a677041-image-registry-private-configuration\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.095424 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.095403    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab82597c-dd5c-45ee-86c7-51768a677041-installation-pull-secrets\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.104129 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.104072    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-bound-sa-token\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.104248 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.104201    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w5b2\" (UniqueName: \"kubernetes.io/projected/bdc1b1df-104d-4e22-9733-71a65b769d25-kube-api-access-8w5b2\") pod \"network-check-source-7cbc45fddc-l26bh\" (UID: \"bdc1b1df-104d-4e22-9733-71a65b769d25\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-l26bh"
Sep 17 17:58:01.104923 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.104887    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2zs6\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-kube-api-access-x2zs6\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.130112 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.130076    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-l26bh"
Sep 17 17:58:01.153593 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.153552    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:01.291030 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.291002    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-flfs2"]
Sep 17 17:58:01.293760 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:58:01.293735    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d65cc5_2f68_45e9_8359_fff61bd12996.slice/crio-386972be00a645a7eb6cb19b3cb72f125af263c71ce4285d760547690ff21569 WatchSource:0}: Error finding container 386972be00a645a7eb6cb19b3cb72f125af263c71ce4285d760547690ff21569: Status 404 returned error can't find the container with id 386972be00a645a7eb6cb19b3cb72f125af263c71ce4285d760547690ff21569
Sep 17 17:58:01.333733 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.333698    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:01.341946 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.341920    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6"]
Sep 17 17:58:01.345111 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:58:01.345084    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cffa332_a0e2_4ec9_aaa4_db8efa6fdbb5.slice/crio-51ecbbd3ed451b0845ad4bc7fbea521b221106071f89118be414df0108ca2ca8 WatchSource:0}: Error finding container 51ecbbd3ed451b0845ad4bc7fbea521b221106071f89118be414df0108ca2ca8: Status 404 returned error can't find the container with id 51ecbbd3ed451b0845ad4bc7fbea521b221106071f89118be414df0108ca2ca8
Sep 17 17:58:01.347533 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.347490    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-tqbd2"]
Sep 17 17:58:01.349742 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:58:01.349721    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c9f091_33e4_4108_beec_d30e38305a30.slice/crio-952fef4a8fd7ddb7399f59e92296e63cefdefb9e8f5c0785eca00e5cf5077059 WatchSource:0}: Error finding container 952fef4a8fd7ddb7399f59e92296e63cefdefb9e8f5c0785eca00e5cf5077059: Status 404 returned error can't find the container with id 952fef4a8fd7ddb7399f59e92296e63cefdefb9e8f5c0785eca00e5cf5077059
Sep 17 17:58:01.403432 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.403401    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-l26bh"]
Sep 17 17:58:01.406107 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:58:01.406078    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc1b1df_104d_4e22_9733_71a65b769d25.slice/crio-0076a1c1f90e82c44eb18b8419eb0e7973da40a183e89a66e54aeb033eb67c9d WatchSource:0}: Error finding container 0076a1c1f90e82c44eb18b8419eb0e7973da40a183e89a66e54aeb033eb67c9d: Status 404 returned error can't find the container with id 0076a1c1f90e82c44eb18b8419eb0e7973da40a183e89a66e54aeb033eb67c9d
Sep 17 17:58:01.594763 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:01.594667    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:01.594922 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:01.594787    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:58:01.594922 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:01.594799    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d8456986-md2f4: secret "image-registry-tls" not found
Sep 17 17:58:01.594922 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:01.594848    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls podName:ab82597c-dd5c-45ee-86c7-51768a677041 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:02.594835141 +0000 UTC m=+52.895657442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls") pod "image-registry-65d8456986-md2f4" (UID: "ab82597c-dd5c-45ee-86c7-51768a677041") : secret "image-registry-tls" not found
Sep 17 17:58:02.068992 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.068954    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"]
Sep 17 17:58:02.071013 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.070990    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"
Sep 17 17:58:02.074317 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.073834    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 17:58:02.074317 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.074085    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9wkxv\""
Sep 17 17:58:02.074317 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.074311    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 17:58:02.090845 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.090818    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"]
Sep 17 17:58:02.199226 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.199190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/15845110-5c22-4b7c-b600-86ec6e146fe1-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-gp5gh\" (UID: \"15845110-5c22-4b7c-b600-86ec6e146fe1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"
Sep 17 17:58:02.199410 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.199232    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gp5gh\" (UID: \"15845110-5c22-4b7c-b600-86ec6e146fe1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"
Sep 17 17:58:02.297112 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.297041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-flfs2" event={"ID":"04d65cc5-2f68-45e9-8359-fff61bd12996","Type":"ContainerStarted","Data":"386972be00a645a7eb6cb19b3cb72f125af263c71ce4285d760547690ff21569"}
Sep 17 17:58:02.299917 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.299890    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/15845110-5c22-4b7c-b600-86ec6e146fe1-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-gp5gh\" (UID: \"15845110-5c22-4b7c-b600-86ec6e146fe1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"
Sep 17 17:58:02.300032 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.299931    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gp5gh\" (UID: \"15845110-5c22-4b7c-b600-86ec6e146fe1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"
Sep 17 17:58:02.300032 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.299973    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-tqbd2" event={"ID":"39c9f091-33e4-4108-beec-d30e38305a30","Type":"ContainerStarted","Data":"952fef4a8fd7ddb7399f59e92296e63cefdefb9e8f5c0785eca00e5cf5077059"}
Sep 17 17:58:02.300174 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:02.300078    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:58:02.300174 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:02.300136    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert podName:15845110-5c22-4b7c-b600-86ec6e146fe1 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:02.8001165 +0000 UTC m=+53.100938801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gp5gh" (UID: "15845110-5c22-4b7c-b600-86ec6e146fe1") : secret "networking-console-plugin-cert" not found
Sep 17 17:58:02.300673 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.300654    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/15845110-5c22-4b7c-b600-86ec6e146fe1-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-gp5gh\" (UID: \"15845110-5c22-4b7c-b600-86ec6e146fe1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"
Sep 17 17:58:02.303366 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.302861    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-l26bh" event={"ID":"bdc1b1df-104d-4e22-9733-71a65b769d25","Type":"ContainerStarted","Data":"b4bb18de517c7cc8ff61613f9d770255593b809d281fac3228b6e9cdcc0e4687"}
Sep 17 17:58:02.303366 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.302904    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-l26bh" event={"ID":"bdc1b1df-104d-4e22-9733-71a65b769d25","Type":"ContainerStarted","Data":"0076a1c1f90e82c44eb18b8419eb0e7973da40a183e89a66e54aeb033eb67c9d"}
Sep 17 17:58:02.307372 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.307243    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6" event={"ID":"8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5","Type":"ContainerStarted","Data":"51ecbbd3ed451b0845ad4bc7fbea521b221106071f89118be414df0108ca2ca8"}
Sep 17 17:58:02.325436 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.325334    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-l26bh" podStartSLOduration=2.325313412 podStartE2EDuration="2.325313412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:02.324364248 +0000 UTC m=+52.625186571" watchObservedRunningTime="2026-09-17 17:58:02.325313412 +0000 UTC m=+52.626135735"
Sep 17 17:58:02.602473 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.602384    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:02.602665 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:02.602622    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:58:02.602665 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:02.602636    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d8456986-md2f4: secret "image-registry-tls" not found
Sep 17 17:58:02.602750 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:02.602685    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls podName:ab82597c-dd5c-45ee-86c7-51768a677041 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:04.602670876 +0000 UTC m=+54.903493177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls") pod "image-registry-65d8456986-md2f4" (UID: "ab82597c-dd5c-45ee-86c7-51768a677041") : secret "image-registry-tls" not found
Sep 17 17:58:02.804553 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:02.804491    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gp5gh\" (UID: \"15845110-5c22-4b7c-b600-86ec6e146fe1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"
Sep 17 17:58:02.804728 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:02.804646    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:58:02.804780 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:02.804728    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert podName:15845110-5c22-4b7c-b600-86ec6e146fe1 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:03.804711431 +0000 UTC m=+54.105533732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gp5gh" (UID: "15845110-5c22-4b7c-b600-86ec6e146fe1") : secret "networking-console-plugin-cert" not found
Sep 17 17:58:03.315295 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:03.315259    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-tqbd2" event={"ID":"39c9f091-33e4-4108-beec-d30e38305a30","Type":"ContainerStarted","Data":"eb2df4a2d352aa7e0141a0e35d16c9dbb79c76d723028bf946bd29dbaff5d751"}
Sep 17 17:58:03.342976 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:03.342892    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-tqbd2" podStartSLOduration=2.297840727 podStartE2EDuration="3.34286972s" totalImagesPullingTime="1.045028993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:03.342327788 +0000 UTC m=+53.643150111" watchObservedRunningTime="2026-09-17 17:58:03.34286972 +0000 UTC m=+53.643692042"
Sep 17 17:58:03.661988 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:03.661911    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:03.812906 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:03.812870    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gp5gh\" (UID: \"15845110-5c22-4b7c-b600-86ec6e146fe1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"
Sep 17 17:58:03.813107 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:03.812994    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:58:03.813107 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:03.813052    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert podName:15845110-5c22-4b7c-b600-86ec6e146fe1 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.813038126 +0000 UTC m=+56.113860427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gp5gh" (UID: "15845110-5c22-4b7c-b600-86ec6e146fe1") : secret "networking-console-plugin-cert" not found
Sep 17 17:58:03.950691 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:03.950593    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:04.320192 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:04.320158    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6" event={"ID":"8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5","Type":"ContainerStarted","Data":"48279015c93d61313a59a0fd58b2cec329f5277bc8b31ad921de76c7262db57a"}
Sep 17 17:58:04.321833 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:04.321805    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-flfs2" event={"ID":"04d65cc5-2f68-45e9-8359-fff61bd12996","Type":"ContainerStarted","Data":"be81f90f065fd9b0628ea938b81a57140eaa645d30843cbbfd0c01edc4ccdb1d"}
Sep 17 17:58:04.363455 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:04.363406    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6" podStartSLOduration=2.132030202 podStartE2EDuration="4.363386156s" totalImagesPullingTime="2.231355954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:04.343280646 +0000 UTC m=+54.644102979" watchObservedRunningTime="2026-09-17 17:58:04.363386156 +0000 UTC m=+54.664208481"
Sep 17 17:58:04.618384 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:04.618300    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:04.618531 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:04.618448    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:58:04.618531 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:04.618467    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d8456986-md2f4: secret "image-registry-tls" not found
Sep 17 17:58:04.618615 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:04.618541    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls podName:ab82597c-dd5c-45ee-86c7-51768a677041 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:08.618507632 +0000 UTC m=+58.919329933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls") pod "image-registry-65d8456986-md2f4" (UID: "ab82597c-dd5c-45ee-86c7-51768a677041") : secret "image-registry-tls" not found
Sep 17 17:58:04.821164 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:04.821086    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:05.069610 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.069568    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:05.111030 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.110963    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-flfs2" podStartSLOduration=2.835116286 podStartE2EDuration="5.110938984s" totalImagesPullingTime="2.275822698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:04.363870883 +0000 UTC m=+54.664693208" watchObservedRunningTime="2026-09-17 17:58:05.110938984 +0000 UTC m=+55.411761308"
Sep 17 17:58:05.112629 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.112603    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-55r82"]
Sep 17 17:58:05.116124 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.116106    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.121051 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.121025    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ssr9g\""
Sep 17 17:58:05.121244 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.121228    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:58:05.124018 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.124002    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:58:05.134149 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.134122    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-55r82"]
Sep 17 17:58:05.203174 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.203147    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:05.223787 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.223758    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b53aee2-867a-44e4-91fa-f894cde05535-data-volume\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.223908 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.223812    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mchjs\" (UniqueName: \"kubernetes.io/projected/4b53aee2-867a-44e4-91fa-f894cde05535-kube-api-access-mchjs\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.223965 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.223909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4b53aee2-867a-44e4-91fa-f894cde05535-crio-socket\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.223965 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.223934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.224052 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.223973    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.324736 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.324669    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.324736 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.324726    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b53aee2-867a-44e4-91fa-f894cde05535-data-volume\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.325140 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.324777    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mchjs\" (UniqueName: \"kubernetes.io/projected/4b53aee2-867a-44e4-91fa-f894cde05535-kube-api-access-mchjs\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.325140 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:05.324802    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:05.325140 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.324839    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4b53aee2-867a-44e4-91fa-f894cde05535-crio-socket\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.325140 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:05.324874    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm podName:4b53aee2-867a-44e4-91fa-f894cde05535 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.824853621 +0000 UTC m=+56.125675939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55r82" (UID: "4b53aee2-867a-44e4-91fa-f894cde05535") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:05.325140 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.324895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.325140 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.324906    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4b53aee2-867a-44e4-91fa-f894cde05535-crio-socket\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.325140 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:05.325014    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:05.325140 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.325053    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b53aee2-867a-44e4-91fa-f894cde05535-data-volume\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.325140 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:05.325071    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls podName:4b53aee2-867a-44e4-91fa-f894cde05535 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.8250557 +0000 UTC m=+56.125878004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55r82" (UID: "4b53aee2-867a-44e4-91fa-f894cde05535") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:05.335678 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.335654    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mchjs\" (UniqueName: \"kubernetes.io/projected/4b53aee2-867a-44e4-91fa-f894cde05535-kube-api-access-mchjs\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.829216 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.829176    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.829216 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.829221    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:05.829418 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:05.829257    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gp5gh\" (UID: \"15845110-5c22-4b7c-b600-86ec6e146fe1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"
Sep 17 17:58:05.829418 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:05.829332    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:05.829418 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:05.829344    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:58:05.829418 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:05.829352    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:05.829418 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:05.829405    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm podName:4b53aee2-867a-44e4-91fa-f894cde05535 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:06.829383265 +0000 UTC m=+57.130205581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55r82" (UID: "4b53aee2-867a-44e4-91fa-f894cde05535") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:05.829597 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:05.829423    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert podName:15845110-5c22-4b7c-b600-86ec6e146fe1 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:09.829415524 +0000 UTC m=+60.130237825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gp5gh" (UID: "15845110-5c22-4b7c-b600-86ec6e146fe1") : secret "networking-console-plugin-cert" not found
Sep 17 17:58:05.829597 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:05.829433    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls podName:4b53aee2-867a-44e4-91fa-f894cde05535 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:06.829428653 +0000 UTC m=+57.130250954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55r82" (UID: "4b53aee2-867a-44e4-91fa-f894cde05535") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:06.837092 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:06.837061    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:06.837092 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:06.837097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:06.837483 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:06.837198    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:06.837483 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:06.837199    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:06.837483 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:06.837255    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm podName:4b53aee2-867a-44e4-91fa-f894cde05535 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:08.837235703 +0000 UTC m=+59.138058005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55r82" (UID: "4b53aee2-867a-44e4-91fa-f894cde05535") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:06.837483 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:06.837268    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls podName:4b53aee2-867a-44e4-91fa-f894cde05535 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:08.837262418 +0000 UTC m=+59.138084719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55r82" (UID: "4b53aee2-867a-44e4-91fa-f894cde05535") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:07.123500 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:07.123394    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:07.569284 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:07.569237    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:07.592967 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:07.592943    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4mlb8_e2c380ad-08a8-4689-b88f-f07b659e6314/dns-node-resolver/0.log"
Sep 17 17:58:07.828411 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:07.828306    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:08.034371 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:08.034338    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:08.600308 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:08.600284    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4vwlx_a8fb6002-f122-4cc8-a73b-3d1e45addd6e/node-ca/0.log"
Sep 17 17:58:08.650337 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:08.650308    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:08.650498 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:08.650467    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:58:08.650498 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:08.650481    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65d8456986-md2f4: secret "image-registry-tls" not found
Sep 17 17:58:08.650596 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:08.650557    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls podName:ab82597c-dd5c-45ee-86c7-51768a677041 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:16.650543269 +0000 UTC m=+66.951365570 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls") pod "image-registry-65d8456986-md2f4" (UID: "ab82597c-dd5c-45ee-86c7-51768a677041") : secret "image-registry-tls" not found
Sep 17 17:58:08.851577 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:08.851460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:08.851577 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:08.851559    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:08.851766 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:08.851628    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:08.851766 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:08.851677    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:08.851766 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:08.851688    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls podName:4b53aee2-867a-44e4-91fa-f894cde05535 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:12.851674408 +0000 UTC m=+63.152496709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55r82" (UID: "4b53aee2-867a-44e4-91fa-f894cde05535") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:08.851766 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:08.851725    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm podName:4b53aee2-867a-44e4-91fa-f894cde05535 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:12.851711732 +0000 UTC m=+63.152534038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55r82" (UID: "4b53aee2-867a-44e4-91fa-f894cde05535") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:09.472661 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:09.472622    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:09.583991 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:09.583967    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jl2b8_bf9e50fc-d402-4599-8278-cc186d5a07ab/serve-healthcheck-canary/0.log"
Sep 17 17:58:09.858625 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:09.858579    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gp5gh\" (UID: \"15845110-5c22-4b7c-b600-86ec6e146fe1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"
Sep 17 17:58:09.858794 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:09.858731    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:58:09.858835 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:09.858797    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert podName:15845110-5c22-4b7c-b600-86ec6e146fe1 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:17.858778387 +0000 UTC m=+68.159600691 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gp5gh" (UID: "15845110-5c22-4b7c-b600-86ec6e146fe1") : secret "networking-console-plugin-cert" not found
Sep 17 17:58:10.232083 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:10.232008    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2v2z"
Sep 17 17:58:10.400479 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:10.400450    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-4bbj6_8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5/kube-storage-version-migrator-operator/0.log"
Sep 17 17:58:10.473718 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:10.473682    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:10.695205 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:10.695168    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:11.055465 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:11.055430    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:11.219917 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:11.219883    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:11.486778 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:11.486679    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:11.664141 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:11.664104    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:11.832124 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:11.832095    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:12.031039 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:12.031003    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:12.481332 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:12.481300    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:12.882930 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:12.882893    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:12.882930 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:12.882939    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:12.883385 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:12.883047    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:12.883385 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:12.883054    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:12.883385 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:12.883109    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm podName:4b53aee2-867a-44e4-91fa-f894cde05535 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:20.883096555 +0000 UTC m=+71.183918861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55r82" (UID: "4b53aee2-867a-44e4-91fa-f894cde05535") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:12.883385 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:12.883120    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls podName:4b53aee2-867a-44e4-91fa-f894cde05535 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:20.883114819 +0000 UTC m=+71.183937120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55r82" (UID: "4b53aee2-867a-44e4-91fa-f894cde05535") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:13.311255 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:13.311222    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:13.721578 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:13.721476    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:13.903773 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:13.903742    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:14.100609 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:14.100584    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:14.193112 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:14.193073    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:58:14.193274 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:14.193226    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:58:14.193318 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:14.193285    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls podName:da415339-7f86-4698-94a7-625f0e5015b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:46.193268585 +0000 UTC m=+96.494090889 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls") pod "dns-default-pbkvq" (UID: "da415339-7f86-4698-94a7-625f0e5015b2") : secret "dns-default-metrics-tls" not found
Sep 17 17:58:14.544359 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:14.544323    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:14.689268 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:14.689234    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:14.924858 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:14.924780    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:15.145227 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:15.145197    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:15.718342 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:15.718305    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:15.883483 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:15.883451    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:16.021439 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:16.021410    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:16.195713 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:16.195684    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:16.340648 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:16.340573    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:16.610537 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:16.610437    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs\") pod \"network-metrics-daemon-xjh98\" (UID: \"3ee045a7-cfc1-40b2-87a8-65bedec89f66\") " pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:58:16.610689 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:16.610607    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:58:16.610727 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:16.610698    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs podName:3ee045a7-cfc1-40b2-87a8-65bedec89f66 nodeName:}" failed. No retries permitted until 2026-09-17 17:59:20.610682082 +0000 UTC m=+130.911504383 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs") pod "network-metrics-daemon-xjh98" (UID: "3ee045a7-cfc1-40b2-87a8-65bedec89f66") : secret "metrics-daemon-secret" not found
Sep 17 17:58:16.711563 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:16.711501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:16.713996 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:16.713963    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls\") pod \"image-registry-65d8456986-md2f4\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") " pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:16.737692 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:16.737657    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-skznp\""
Sep 17 17:58:16.745872 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:16.745837    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:17.031676 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:17.031643    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65d8456986-md2f4"]
Sep 17 17:58:17.036287 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:58:17.036264    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab82597c_dd5c_45ee_86c7_51768a677041.slice/crio-ca1af2a4fd2ef8ffda69e9ac34c6b33dfeec76e16cdd0e9ac52c7cbf88063a21 WatchSource:0}: Error finding container ca1af2a4fd2ef8ffda69e9ac34c6b33dfeec76e16cdd0e9ac52c7cbf88063a21: Status 404 returned error can't find the container with id ca1af2a4fd2ef8ffda69e9ac34c6b33dfeec76e16cdd0e9ac52c7cbf88063a21
Sep 17 17:58:17.339441 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:17.339360    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:17.368133 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:17.368093    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d8456986-md2f4" event={"ID":"ab82597c-dd5c-45ee-86c7-51768a677041","Type":"ContainerStarted","Data":"de7380864942b9ee97b475c33ba0ae2aef78acc8783e379feb991eab81295efc"}
Sep 17 17:58:17.368305 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:17.368201    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:17.368305 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:17.368223    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d8456986-md2f4" event={"ID":"ab82597c-dd5c-45ee-86c7-51768a677041","Type":"ContainerStarted","Data":"ca1af2a4fd2ef8ffda69e9ac34c6b33dfeec76e16cdd0e9ac52c7cbf88063a21"}
Sep 17 17:58:17.407176 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:17.407106    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65d8456986-md2f4" podStartSLOduration=17.407085183 podStartE2EDuration="17.407085183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:17.403650982 +0000 UTC m=+67.704473305" watchObservedRunningTime="2026-09-17 17:58:17.407085183 +0000 UTC m=+67.707907506"
Sep 17 17:58:17.569209 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:17.569173    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:17.772721 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:17.772685    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:17.920358 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:17.920313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gp5gh\" (UID: \"15845110-5c22-4b7c-b600-86ec6e146fe1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"
Sep 17 17:58:17.920571 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:17.920452    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:58:17.920571 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:17.920541    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert podName:15845110-5c22-4b7c-b600-86ec6e146fe1 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:33.920502407 +0000 UTC m=+84.221324713 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gp5gh" (UID: "15845110-5c22-4b7c-b600-86ec6e146fe1") : secret "networking-console-plugin-cert" not found
Sep 17 17:58:17.955067 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:17.955039    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:18.155340 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:18.155267    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:18.915528 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:18.915489    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:19.104424 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:19.104369    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:19.504146 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:19.504094    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:19.719790 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:19.719758    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:20.088155 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:20.088125    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:20.541615 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:20.541585    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:20.723482 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:20.723368    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:20.825274 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:20.825192    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:20.943448 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:20.943403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:20.943634 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:20.943459    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:20.943634 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:20.943559    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:20.943634 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:58:20.943631    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls podName:4b53aee2-867a-44e4-91fa-f894cde05535 nodeName:}" failed. No retries permitted until 2026-09-17 17:58:36.943613071 +0000 UTC m=+87.244435373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55r82" (UID: "4b53aee2-867a-44e4-91fa-f894cde05535") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:20.943996 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:20.943979    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b53aee2-867a-44e4-91fa-f894cde05535-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:21.304974 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:21.304947    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:21.514589 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:21.514548    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:21.736897 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:21.736829    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:21.921501 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:21.921458    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:22.271679 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:22.271649    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kb4ks"
Sep 17 17:58:29.382284 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:29.382250    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:33.939074 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:33.939034    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gp5gh\" (UID: \"15845110-5c22-4b7c-b600-86ec6e146fe1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"
Sep 17 17:58:33.941471 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:33.941450    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15845110-5c22-4b7c-b600-86ec6e146fe1-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gp5gh\" (UID: \"15845110-5c22-4b7c-b600-86ec6e146fe1\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"
Sep 17 17:58:34.184637 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:34.184596    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9wkxv\""
Sep 17 17:58:34.192113 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:34.192055    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"
Sep 17 17:58:34.465883 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:34.465849    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh"]
Sep 17 17:58:34.467776 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:58:34.467747    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15845110_5c22_4b7c_b600_86ec6e146fe1.slice/crio-dcd483642b421e81c1621ee38b8266bf27b7bb980f1969e5ab82e0332129b2ed WatchSource:0}: Error finding container dcd483642b421e81c1621ee38b8266bf27b7bb980f1969e5ab82e0332129b2ed: Status 404 returned error can't find the container with id dcd483642b421e81c1621ee38b8266bf27b7bb980f1969e5ab82e0332129b2ed
Sep 17 17:58:35.396090 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:35.396054    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:35.423861 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:35.423829    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh" event={"ID":"15845110-5c22-4b7c-b600-86ec6e146fe1","Type":"ContainerStarted","Data":"dcd483642b421e81c1621ee38b8266bf27b7bb980f1969e5ab82e0332129b2ed"}
Sep 17 17:58:35.520391 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:35.520353    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:35.691361 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:35.691283    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:36.428183 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:36.428145    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh" event={"ID":"15845110-5c22-4b7c-b600-86ec6e146fe1","Type":"ContainerStarted","Data":"d979de4dcbe6f03579b11838885e5bc7c0d32a736bfe5b4eb7bdec0a6522a2bd"}
Sep 17 17:58:36.446478 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:36.446433    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gp5gh" podStartSLOduration=33.436022768 podStartE2EDuration="34.446417497s" totalImagesPullingTime="1.010394729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:36.445214145 +0000 UTC m=+86.746036457" watchObservedRunningTime="2026-09-17 17:58:36.446417497 +0000 UTC m=+86.747239822"
Sep 17 17:58:36.750718 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:36.750637    2574 patch_prober.go:28] interesting pod/image-registry-65d8456986-md2f4 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:58:36.750718 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:36.750693    2574 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65d8456986-md2f4" podUID="ab82597c-dd5c-45ee-86c7-51768a677041" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:36.963501 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:36.963467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:36.965880 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:36.965863    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b53aee2-867a-44e4-91fa-f894cde05535-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55r82\" (UID: \"4b53aee2-867a-44e4-91fa-f894cde05535\") " pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:37.226653 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:37.226623    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ssr9g\""
Sep 17 17:58:37.235282 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:37.235259    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-55r82"
Sep 17 17:58:37.537733 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:37.537702    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-55r82"]
Sep 17 17:58:37.540962 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:58:37.540932    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b53aee2_867a_44e4_91fa_f894cde05535.slice/crio-daaf11e33b826c8553760eb9686597b6cb413d3a0a1aa3bb2a2318193f1e6caa WatchSource:0}: Error finding container daaf11e33b826c8553760eb9686597b6cb413d3a0a1aa3bb2a2318193f1e6caa: Status 404 returned error can't find the container with id daaf11e33b826c8553760eb9686597b6cb413d3a0a1aa3bb2a2318193f1e6caa
Sep 17 17:58:38.103505 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.103480    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8l6qg"]
Sep 17 17:58:38.105338 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.105325    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8l6qg"
Sep 17 17:58:38.110886 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.110871    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 17:58:38.110953 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.110872    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-48rtt\""
Sep 17 17:58:38.158216 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.158192    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65d8456986-md2f4"]
Sep 17 17:58:38.162237 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.162216    2574 patch_prober.go:28] interesting pod/image-registry-65d8456986-md2f4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:58:38.162345 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.162263    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-65d8456986-md2f4" podUID="ab82597c-dd5c-45ee-86c7-51768a677041" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:58:38.171721 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.171693    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/043673ea-e295-4b8a-a582-3af684b752b1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-8l6qg\" (UID: \"043673ea-e295-4b8a-a582-3af684b752b1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8l6qg"
Sep 17 17:58:38.272107 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.272075    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/043673ea-e295-4b8a-a582-3af684b752b1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-8l6qg\" (UID: \"043673ea-e295-4b8a-a582-3af684b752b1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8l6qg"
Sep 17 17:58:38.274597 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.274576    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/043673ea-e295-4b8a-a582-3af684b752b1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-8l6qg\" (UID: \"043673ea-e295-4b8a-a582-3af684b752b1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8l6qg"
Sep 17 17:58:38.279982 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.279961    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8l6qg"]
Sep 17 17:58:38.365857 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.365786    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-795f647d56-d6k2l"]
Sep 17 17:58:38.367841 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.367826    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.368905 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.368883    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:38.411949 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.411923    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-795f647d56-d6k2l"]
Sep 17 17:58:38.420944 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.420923    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8l6qg"
Sep 17 17:58:38.436878 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.436852    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-55r82" event={"ID":"4b53aee2-867a-44e4-91fa-f894cde05535","Type":"ContainerStarted","Data":"dd2b84a5e87d1fef3308c4f6aee35a78cd4d4173a163c36e373251fca9428380"}
Sep 17 17:58:38.436983 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.436905    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-55r82" event={"ID":"4b53aee2-867a-44e4-91fa-f894cde05535","Type":"ContainerStarted","Data":"157a7eb8e4c23efd55118147246bafb6486d416dc45d6d6be14fa0ac6447a73f"}
Sep 17 17:58:38.436983 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.436924    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-55r82" event={"ID":"4b53aee2-867a-44e4-91fa-f894cde05535","Type":"ContainerStarted","Data":"daaf11e33b826c8553760eb9686597b6cb413d3a0a1aa3bb2a2318193f1e6caa"}
Sep 17 17:58:38.477149 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.476735    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb0a4576-e4c8-423e-b504-1916fb761cdc-installation-pull-secrets\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.477149 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.476788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nrt9\" (UniqueName: \"kubernetes.io/projected/eb0a4576-e4c8-423e-b504-1916fb761cdc-kube-api-access-2nrt9\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.477149 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.476821    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb0a4576-e4c8-423e-b504-1916fb761cdc-registry-certificates\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.477149 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.476844    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb0a4576-e4c8-423e-b504-1916fb761cdc-trusted-ca\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.477149 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.476872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb0a4576-e4c8-423e-b504-1916fb761cdc-registry-tls\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.477149 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.476930    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb0a4576-e4c8-423e-b504-1916fb761cdc-image-registry-private-configuration\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.477149 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.476987    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb0a4576-e4c8-423e-b504-1916fb761cdc-bound-sa-token\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.477149 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.477028    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb0a4576-e4c8-423e-b504-1916fb761cdc-ca-trust-extracted\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.511476 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.511438    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:38.577435 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.577398    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb0a4576-e4c8-423e-b504-1916fb761cdc-registry-certificates\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.577435 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.577446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb0a4576-e4c8-423e-b504-1916fb761cdc-trusted-ca\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.577937 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.577473    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb0a4576-e4c8-423e-b504-1916fb761cdc-registry-tls\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.577937 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.577538    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb0a4576-e4c8-423e-b504-1916fb761cdc-image-registry-private-configuration\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.577937 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.577587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb0a4576-e4c8-423e-b504-1916fb761cdc-bound-sa-token\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.577937 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.577629    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb0a4576-e4c8-423e-b504-1916fb761cdc-ca-trust-extracted\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.577937 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.577665    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb0a4576-e4c8-423e-b504-1916fb761cdc-installation-pull-secrets\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.577937 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.577695    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nrt9\" (UniqueName: \"kubernetes.io/projected/eb0a4576-e4c8-423e-b504-1916fb761cdc-kube-api-access-2nrt9\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.578318 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.578263    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb0a4576-e4c8-423e-b504-1916fb761cdc-ca-trust-extracted\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.578667 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.578428    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb0a4576-e4c8-423e-b504-1916fb761cdc-registry-certificates\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.578667 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.578467    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb0a4576-e4c8-423e-b504-1916fb761cdc-trusted-ca\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.580660 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.580635    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb0a4576-e4c8-423e-b504-1916fb761cdc-image-registry-private-configuration\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.580964 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.580942    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb0a4576-e4c8-423e-b504-1916fb761cdc-installation-pull-secrets\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.581063 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.581049    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb0a4576-e4c8-423e-b504-1916fb761cdc-registry-tls\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.597186 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.597156    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nrt9\" (UniqueName: \"kubernetes.io/projected/eb0a4576-e4c8-423e-b504-1916fb761cdc-kube-api-access-2nrt9\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.597993 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.597969    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb0a4576-e4c8-423e-b504-1916fb761cdc-bound-sa-token\") pod \"image-registry-795f647d56-d6k2l\" (UID: \"eb0a4576-e4c8-423e-b504-1916fb761cdc\") " pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.625051 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.624935    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:38.676177 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.676135    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:38.702684 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.702648    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8l6qg"]
Sep 17 17:58:38.707488 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:58:38.707462    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043673ea_e295_4b8a_a582_3af684b752b1.slice/crio-7c34ae4f64e0423ea07e2d5c62d868fca5fc3e44ffe36abc88d50e5511c6d04d WatchSource:0}: Error finding container 7c34ae4f64e0423ea07e2d5c62d868fca5fc3e44ffe36abc88d50e5511c6d04d: Status 404 returned error can't find the container with id 7c34ae4f64e0423ea07e2d5c62d868fca5fc3e44ffe36abc88d50e5511c6d04d
Sep 17 17:58:38.975687 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:38.975614    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-795f647d56-d6k2l"]
Sep 17 17:58:39.281568 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:58:39.281536    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0a4576_e4c8_423e_b504_1916fb761cdc.slice/crio-d2b0ff70efb1769e5ebf4ce2da01b3b536087a4dde44636894b5c3c33d0eb67a WatchSource:0}: Error finding container d2b0ff70efb1769e5ebf4ce2da01b3b536087a4dde44636894b5c3c33d0eb67a: Status 404 returned error can't find the container with id d2b0ff70efb1769e5ebf4ce2da01b3b536087a4dde44636894b5c3c33d0eb67a
Sep 17 17:58:39.442346 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:39.442313    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-795f647d56-d6k2l" event={"ID":"eb0a4576-e4c8-423e-b504-1916fb761cdc","Type":"ContainerStarted","Data":"00be6e9ff9aa7e5420b2c6dd40593869d766cd651c1553ae7b6f5d53b7d3d5f1"}
Sep 17 17:58:39.442553 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:39.442367    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-795f647d56-d6k2l" event={"ID":"eb0a4576-e4c8-423e-b504-1916fb761cdc","Type":"ContainerStarted","Data":"d2b0ff70efb1769e5ebf4ce2da01b3b536087a4dde44636894b5c3c33d0eb67a"}
Sep 17 17:58:39.442553 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:39.442391    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:58:39.444940 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:39.444916    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-55r82" event={"ID":"4b53aee2-867a-44e4-91fa-f894cde05535","Type":"ContainerStarted","Data":"52283f7ad3e01c3d2ea833498cdb2b4d289d36b0752ad1b73cc67e8fd2f78266"}
Sep 17 17:58:39.446219 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:39.446195    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8l6qg" event={"ID":"043673ea-e295-4b8a-a582-3af684b752b1","Type":"ContainerStarted","Data":"7c34ae4f64e0423ea07e2d5c62d868fca5fc3e44ffe36abc88d50e5511c6d04d"}
Sep 17 17:58:39.468977 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:39.468924    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-795f647d56-d6k2l" podStartSLOduration=1.4689035449999999 podStartE2EDuration="1.468903545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:39.467166343 +0000 UTC m=+89.767988667" watchObservedRunningTime="2026-09-17 17:58:39.468903545 +0000 UTC m=+89.769725869"
Sep 17 17:58:39.494471 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:39.494424    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-55r82" podStartSLOduration=32.857416941 podStartE2EDuration="34.494406683s" totalImagesPullingTime="1.636989742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:58:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:39.493907878 +0000 UTC m=+89.794730200" watchObservedRunningTime="2026-09-17 17:58:39.494406683 +0000 UTC m=+89.795229006"
Sep 17 17:58:40.453160 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:40.453119    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8l6qg" event={"ID":"043673ea-e295-4b8a-a582-3af684b752b1","Type":"ContainerStarted","Data":"54501674dfe3f5f17049dea70cde365f508717f9af48fce78f257ee00ea281eb"}
Sep 17 17:58:40.453637 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:40.453354    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8l6qg"
Sep 17 17:58:40.457887 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:40.457859    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8l6qg"
Sep 17 17:58:40.502273 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:40.502221    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8l6qg" podStartSLOduration=1.363466788 podStartE2EDuration="2.502207865s" totalImagesPullingTime="1.138741077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:40.478575608 +0000 UTC m=+90.779397932" watchObservedRunningTime="2026-09-17 17:58:40.502207865 +0000 UTC m=+90.803030188"
Sep 17 17:58:41.605750 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:41.605718    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:41.744002 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:41.743965    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:41.935016 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:41.934926    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:42.046837 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:42.046802    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:43.813673 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:43.813633    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:46.138296 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.138264    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:46.234804 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.234779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:58:46.237099 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.237073    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da415339-7f86-4698-94a7-625f0e5015b2-metrics-tls\") pod \"dns-default-pbkvq\" (UID: \"da415339-7f86-4698-94a7-625f0e5015b2\") " pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:58:46.305779 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.305745    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:46.310269 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.310255    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4ffrq\""
Sep 17 17:58:46.318645 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.318628    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:58:46.570581 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.570540    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:46.610047 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.610027    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pbkvq"]
Sep 17 17:58:46.612509 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:58:46.612482    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda415339_7f86_4698_94a7_625f0e5015b2.slice/crio-5045ba988c20d2281e026390b5452bdbd5df242031d9173e37605adad9c0a42a WatchSource:0}: Error finding container 5045ba988c20d2281e026390b5452bdbd5df242031d9173e37605adad9c0a42a: Status 404 returned error can't find the container with id 5045ba988c20d2281e026390b5452bdbd5df242031d9173e37605adad9c0a42a
Sep 17 17:58:46.721571 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.721539    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mpx8b"]
Sep 17 17:58:46.725611 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.725596    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.729481 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.729461    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:58:46.729657 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.729598    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:58:46.742135 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.742108    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:58:46.748530 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.748499    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:58:46.748656 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.748639    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:58:46.748725 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.748669    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:58:46.748863 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.748849    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2w4t8\""
Sep 17 17:58:46.786558 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.786531    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:46.840233 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.840154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.840233 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.840193    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmzh7\" (UniqueName: \"kubernetes.io/projected/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-kube-api-access-lmzh7\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.840233 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.840220    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-root\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.840482 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.840273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-sys\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.840482 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.840303    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-tls\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.840482 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.840360    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-metrics-client-ca\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.840482 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.840400    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-textfile\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.840482 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.840426    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.840482 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.840464    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-wtmp\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.941784 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.941755    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-metrics-client-ca\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.942094 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.941790    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-textfile\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.942094 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.941808    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.942094 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.941839    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-wtmp\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.942094 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.941880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.942094 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.941907    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmzh7\" (UniqueName: \"kubernetes.io/projected/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-kube-api-access-lmzh7\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.942094 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.941932    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-root\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.942094 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.941985    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-root\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.942094 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.942020    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-sys\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.942094 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.942054    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-tls\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.942094 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.942060    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-wtmp\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.942660 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.942129    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-sys\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.942660 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.942230    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-textfile\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.942660 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.942450    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-metrics-client-ca\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.942660 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.942453    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.944530 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.944493    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-tls\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.944616 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.944549    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:46.962294 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.962258    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:46.994923 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:46.994903    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmzh7\" (UniqueName: \"kubernetes.io/projected/0cecc0ad-f2e1-488b-a2f7-d46f669e8012-kube-api-access-lmzh7\") pod \"node-exporter-mpx8b\" (UID: \"0cecc0ad-f2e1-488b-a2f7-d46f669e8012\") " pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:47.035003 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:47.034975    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mpx8b"
Sep 17 17:58:47.047082 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:58:47.047044    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cecc0ad_f2e1_488b_a2f7_d46f669e8012.slice/crio-c722ee7e6d5fb520419d965443a7457de94b3f7e0016b4545259a167d99b2a5b WatchSource:0}: Error finding container c722ee7e6d5fb520419d965443a7457de94b3f7e0016b4545259a167d99b2a5b: Status 404 returned error can't find the container with id c722ee7e6d5fb520419d965443a7457de94b3f7e0016b4545259a167d99b2a5b
Sep 17 17:58:47.479807 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:47.479766    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpx8b" event={"ID":"0cecc0ad-f2e1-488b-a2f7-d46f669e8012","Type":"ContainerStarted","Data":"c722ee7e6d5fb520419d965443a7457de94b3f7e0016b4545259a167d99b2a5b"}
Sep 17 17:58:47.481531 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:47.481482    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pbkvq" event={"ID":"da415339-7f86-4698-94a7-625f0e5015b2","Type":"ContainerStarted","Data":"5045ba988c20d2281e026390b5452bdbd5df242031d9173e37605adad9c0a42a"}
Sep 17 17:58:47.537953 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:47.537889    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:47.819045 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:47.819009    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:48.098538 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:48.098447    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:48.164283 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:48.164252    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:58:48.384109 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:48.383744    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:48.487013 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:48.486980    2574 generic.go:356] "Generic (PLEG): container finished" podID="0cecc0ad-f2e1-488b-a2f7-d46f669e8012" containerID="a9c4efefbf24a19184f22c21b449b9341deb872777b2be9528ffcbba6c439792" exitCode=0
Sep 17 17:58:48.487367 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:48.487068    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpx8b" event={"ID":"0cecc0ad-f2e1-488b-a2f7-d46f669e8012","Type":"ContainerDied","Data":"a9c4efefbf24a19184f22c21b449b9341deb872777b2be9528ffcbba6c439792"}
Sep 17 17:58:48.488916 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:48.488882    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pbkvq" event={"ID":"da415339-7f86-4698-94a7-625f0e5015b2","Type":"ContainerStarted","Data":"086494d618e7c853d6ba214e886eb8731cc86902d9cb9b5ed399f30244035cce"}
Sep 17 17:58:48.489008 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:48.488931    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pbkvq" event={"ID":"da415339-7f86-4698-94a7-625f0e5015b2","Type":"ContainerStarted","Data":"767e757911885601b0a286b191e5aee940b27dbabc8ea8d0f3880e5b3a275313"}
Sep 17 17:58:48.489008 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:48.488978    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:58:48.580253 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:48.580230    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:48.587939 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:48.587904    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pbkvq" podStartSLOduration=64.939427213 podStartE2EDuration="1m6.587891505s" totalImagesPullingTime="1.648464292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:48.587809863 +0000 UTC m=+98.888632187" watchObservedRunningTime="2026-09-17 17:58:48.587891505 +0000 UTC m=+98.888713827"
Sep 17 17:58:48.695338 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:48.695259    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:49.493955 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:49.493910    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpx8b" event={"ID":"0cecc0ad-f2e1-488b-a2f7-d46f669e8012","Type":"ContainerStarted","Data":"0c4f9c7d0a2e2137c9885173560792ac7261bd27c2452640247f79a4f96aca46"}
Sep 17 17:58:49.494407 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:49.493997    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpx8b" event={"ID":"0cecc0ad-f2e1-488b-a2f7-d46f669e8012","Type":"ContainerStarted","Data":"76e37cd9214d5993017760f7bb95d8cb42f0125a1dbe6fbb9e28e25e7a6d6b41"}
Sep 17 17:58:49.552171 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:49.552121    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mpx8b" podStartSLOduration=2.259893468 podStartE2EDuration="3.552106841s" totalImagesPullingTime="1.2137678s" totalInitContainerRuntime="78.445573ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:49.550798979 +0000 UTC m=+99.851621301" watchObservedRunningTime="2026-09-17 17:58:49.552106841 +0000 UTC m=+99.852929164"
Sep 17 17:58:49.901723 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:49.901686    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:50.079034 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:50.079001    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:50.251663 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:50.251579    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:50.374284 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:50.374239    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:50.621343 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:50.621305    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:50.802409 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:50.802369    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:51.066902 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.066867    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:51.078429 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.078399    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh"]
Sep 17 17:58:51.081245 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.081230    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh"
Sep 17 17:58:51.088936 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.088919    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 17:58:51.089134 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.089122    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8q99t\""
Sep 17 17:58:51.103296 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.103272    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh"]
Sep 17 17:58:51.179527 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.179486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-sm8vh\" (UID: \"12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh"
Sep 17 17:58:51.179722 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.179554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq6zb\" (UniqueName: \"kubernetes.io/projected/12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a-kube-api-access-nq6zb\") pod \"monitoring-plugin-5984459ff8-sm8vh\" (UID: \"12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh"
Sep 17 17:58:51.224081 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.224049    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:51.280791 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.280762    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-sm8vh\" (UID: \"12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh"
Sep 17 17:58:51.280926 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.280806    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq6zb\" (UniqueName: \"kubernetes.io/projected/12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a-kube-api-access-nq6zb\") pod \"monitoring-plugin-5984459ff8-sm8vh\" (UID: \"12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh"
Sep 17 17:58:51.283280 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.283258    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-sm8vh\" (UID: \"12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh"
Sep 17 17:58:51.290122 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.290096    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq6zb\" (UniqueName: \"kubernetes.io/projected/12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a-kube-api-access-nq6zb\") pod \"monitoring-plugin-5984459ff8-sm8vh\" (UID: \"12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh"
Sep 17 17:58:51.389949 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.389871    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh"
Sep 17 17:58:51.679101 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:51.679034    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh"]
Sep 17 17:58:51.682206 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:58:51.682179    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12abcf49_2ce7_4fa4_ac1a_b8f6e3955e2a.slice/crio-1d15fb5e99f1cbe938ececeb03d923f9fe2c73e2177c2c8c88e6e055d8fad201 WatchSource:0}: Error finding container 1d15fb5e99f1cbe938ececeb03d923f9fe2c73e2177c2c8c88e6e055d8fad201: Status 404 returned error can't find the container with id 1d15fb5e99f1cbe938ececeb03d923f9fe2c73e2177c2c8c88e6e055d8fad201
Sep 17 17:58:52.508056 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:52.508018    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh" event={"ID":"12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a","Type":"ContainerStarted","Data":"1d15fb5e99f1cbe938ececeb03d923f9fe2c73e2177c2c8c88e6e055d8fad201"}
Sep 17 17:58:52.924286 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:52.924255    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:53.051932 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:53.051892    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:53.167673 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:53.167599    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:53.512974 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:53.512936    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh" event={"ID":"12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a","Type":"ContainerStarted","Data":"c6cdd4863e7cfdbd41dc6c8ea4a6d098d74f865162295f30e2b5ff979ea762d9"}
Sep 17 17:58:53.513148 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:53.513111    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh"
Sep 17 17:58:53.517697 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:53.517671    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh"
Sep 17 17:58:53.546309 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:53.546242    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-sm8vh" podStartSLOduration=1.319718312 podStartE2EDuration="2.54622816s" totalImagesPullingTime="1.226509848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:53.545107238 +0000 UTC m=+103.845929563" watchObservedRunningTime="2026-09-17 17:58:53.54622816 +0000 UTC m=+103.847050482"
Sep 17 17:58:53.686564 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:53.686527    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:58.222761 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.222722    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:58.226475 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.226453    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-8qvts"]
Sep 17 17:58:58.229575 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.229559    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-8qvts"
Sep 17 17:58:58.239530 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.239495    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wvjc2\""
Sep 17 17:58:58.239652 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.239610    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:58:58.239652 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.239638    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:58:58.339528 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.339488    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854xr\" (UniqueName: \"kubernetes.io/projected/d79f9b27-81c4-47aa-b378-ed1d73cd2b03-kube-api-access-854xr\") pod \"downloads-6594f4c75-8qvts\" (UID: \"d79f9b27-81c4-47aa-b378-ed1d73cd2b03\") " pod="openshift-console/downloads-6594f4c75-8qvts"
Sep 17 17:58:58.342054 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.342017    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-8qvts"]
Sep 17 17:58:58.346767 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.346742    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:58.440374 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.440340    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854xr\" (UniqueName: \"kubernetes.io/projected/d79f9b27-81c4-47aa-b378-ed1d73cd2b03-kube-api-access-854xr\") pod \"downloads-6594f4c75-8qvts\" (UID: \"d79f9b27-81c4-47aa-b378-ed1d73cd2b03\") " pod="openshift-console/downloads-6594f4c75-8qvts"
Sep 17 17:58:58.460497 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.460475    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-854xr\" (UniqueName: \"kubernetes.io/projected/d79f9b27-81c4-47aa-b378-ed1d73cd2b03-kube-api-access-854xr\") pod \"downloads-6594f4c75-8qvts\" (UID: \"d79f9b27-81c4-47aa-b378-ed1d73cd2b03\") " pod="openshift-console/downloads-6594f4c75-8qvts"
Sep 17 17:58:58.496096 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.496045    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pbkvq"
Sep 17 17:58:58.528361 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.528329    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:58.537268 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.537247    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-8qvts"
Sep 17 17:58:58.693248 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.693215    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:58.886114 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.886089    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:58.898228 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:58.898200    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-8qvts"]
Sep 17 17:58:58.901144 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:58:58.901125    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79f9b27_81c4_47aa_b378_ed1d73cd2b03.slice/crio-aaf6d2a66c31f76fd5953c9098604faa39d9ee0ff1fcd473ecb3ac125e943932 WatchSource:0}: Error finding container aaf6d2a66c31f76fd5953c9098604faa39d9ee0ff1fcd473ecb3ac125e943932: Status 404 returned error can't find the container with id aaf6d2a66c31f76fd5953c9098604faa39d9ee0ff1fcd473ecb3ac125e943932
Sep 17 17:58:59.533246 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:59.533210    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-8qvts" event={"ID":"d79f9b27-81c4-47aa-b378-ed1d73cd2b03","Type":"ContainerStarted","Data":"aaf6d2a66c31f76fd5953c9098604faa39d9ee0ff1fcd473ecb3ac125e943932"}
Sep 17 17:58:59.789895 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:58:59.789808    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:00.002745 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:00.002718    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:00.101196 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:00.101099    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:00.202151 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:00.202119    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:00.457672 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:00.457600    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-795f647d56-d6k2l"
Sep 17 17:59:00.660111 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:00.660078    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:00.764610 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:00.764568    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:00.843282 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:00.843249    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:02.436573 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:02.436541    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:02.520668 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:02.520634    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:03.176630 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.176596    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65d8456986-md2f4" podUID="ab82597c-dd5c-45ee-86c7-51768a677041" containerName="registry" containerID="cri-o://de7380864942b9ee97b475c33ba0ae2aef78acc8783e379feb991eab81295efc" gracePeriod=30
Sep 17 17:59:03.428244 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.428185    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:59:03.551724 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.551689    2574 generic.go:356] "Generic (PLEG): container finished" podID="ab82597c-dd5c-45ee-86c7-51768a677041" containerID="de7380864942b9ee97b475c33ba0ae2aef78acc8783e379feb991eab81295efc" exitCode=0
Sep 17 17:59:03.552134 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.551736    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d8456986-md2f4"
Sep 17 17:59:03.552134 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.551777    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d8456986-md2f4" event={"ID":"ab82597c-dd5c-45ee-86c7-51768a677041","Type":"ContainerDied","Data":"de7380864942b9ee97b475c33ba0ae2aef78acc8783e379feb991eab81295efc"}
Sep 17 17:59:03.552134 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.551862    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d8456986-md2f4" event={"ID":"ab82597c-dd5c-45ee-86c7-51768a677041","Type":"ContainerDied","Data":"ca1af2a4fd2ef8ffda69e9ac34c6b33dfeec76e16cdd0e9ac52c7cbf88063a21"}
Sep 17 17:59:03.552134 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.551892    2574 scope.go:117] "RemoveContainer" containerID="de7380864942b9ee97b475c33ba0ae2aef78acc8783e379feb991eab81295efc"
Sep 17 17:59:03.561262 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.561239    2574 scope.go:117] "RemoveContainer" containerID="de7380864942b9ee97b475c33ba0ae2aef78acc8783e379feb991eab81295efc"
Sep 17 17:59:03.561576 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:59:03.561551    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7380864942b9ee97b475c33ba0ae2aef78acc8783e379feb991eab81295efc\": container with ID starting with de7380864942b9ee97b475c33ba0ae2aef78acc8783e379feb991eab81295efc not found: ID does not exist" containerID="de7380864942b9ee97b475c33ba0ae2aef78acc8783e379feb991eab81295efc"
Sep 17 17:59:03.561666 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.561589    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7380864942b9ee97b475c33ba0ae2aef78acc8783e379feb991eab81295efc"} err="failed to get container status \"de7380864942b9ee97b475c33ba0ae2aef78acc8783e379feb991eab81295efc\": rpc error: code = NotFound desc = could not find container \"de7380864942b9ee97b475c33ba0ae2aef78acc8783e379feb991eab81295efc\": container with ID starting with de7380864942b9ee97b475c33ba0ae2aef78acc8783e379feb991eab81295efc not found: ID does not exist"
Sep 17 17:59:03.585335 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.585306    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab82597c-dd5c-45ee-86c7-51768a677041-ca-trust-extracted\") pod \"ab82597c-dd5c-45ee-86c7-51768a677041\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") "
Sep 17 17:59:03.585462 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.585383    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2zs6\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-kube-api-access-x2zs6\") pod \"ab82597c-dd5c-45ee-86c7-51768a677041\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") "
Sep 17 17:59:03.585462 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.585416    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab82597c-dd5c-45ee-86c7-51768a677041-registry-certificates\") pod \"ab82597c-dd5c-45ee-86c7-51768a677041\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") "
Sep 17 17:59:03.585462 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.585435    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-bound-sa-token\") pod \"ab82597c-dd5c-45ee-86c7-51768a677041\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") "
Sep 17 17:59:03.585635 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.585470    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls\") pod \"ab82597c-dd5c-45ee-86c7-51768a677041\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") "
Sep 17 17:59:03.585983 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.585798    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab82597c-dd5c-45ee-86c7-51768a677041-trusted-ca\") pod \"ab82597c-dd5c-45ee-86c7-51768a677041\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") "
Sep 17 17:59:03.585983 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.585854    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab82597c-dd5c-45ee-86c7-51768a677041-installation-pull-secrets\") pod \"ab82597c-dd5c-45ee-86c7-51768a677041\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") "
Sep 17 17:59:03.585983 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.585872    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab82597c-dd5c-45ee-86c7-51768a677041-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ab82597c-dd5c-45ee-86c7-51768a677041" (UID: "ab82597c-dd5c-45ee-86c7-51768a677041"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:03.585983 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.585919    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab82597c-dd5c-45ee-86c7-51768a677041-image-registry-private-configuration\") pod \"ab82597c-dd5c-45ee-86c7-51768a677041\" (UID: \"ab82597c-dd5c-45ee-86c7-51768a677041\") "
Sep 17 17:59:03.586212 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.586126    2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab82597c-dd5c-45ee-86c7-51768a677041-registry-certificates\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 17:59:03.586285 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.586247    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab82597c-dd5c-45ee-86c7-51768a677041-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ab82597c-dd5c-45ee-86c7-51768a677041" (UID: "ab82597c-dd5c-45ee-86c7-51768a677041"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:03.588397 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.588370    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-kube-api-access-x2zs6" (OuterVolumeSpecName: "kube-api-access-x2zs6") pod "ab82597c-dd5c-45ee-86c7-51768a677041" (UID: "ab82597c-dd5c-45ee-86c7-51768a677041"). InnerVolumeSpecName "kube-api-access-x2zs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:03.588397 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.588390    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ab82597c-dd5c-45ee-86c7-51768a677041" (UID: "ab82597c-dd5c-45ee-86c7-51768a677041"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:03.588706 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.588680    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab82597c-dd5c-45ee-86c7-51768a677041-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ab82597c-dd5c-45ee-86c7-51768a677041" (UID: "ab82597c-dd5c-45ee-86c7-51768a677041"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:03.588983 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.588948    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab82597c-dd5c-45ee-86c7-51768a677041-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ab82597c-dd5c-45ee-86c7-51768a677041" (UID: "ab82597c-dd5c-45ee-86c7-51768a677041"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:03.589077 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.589004    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ab82597c-dd5c-45ee-86c7-51768a677041" (UID: "ab82597c-dd5c-45ee-86c7-51768a677041"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:03.596440 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.596413    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab82597c-dd5c-45ee-86c7-51768a677041-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ab82597c-dd5c-45ee-86c7-51768a677041" (UID: "ab82597c-dd5c-45ee-86c7-51768a677041"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:03.686813 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.686727    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2zs6\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-kube-api-access-x2zs6\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 17:59:03.686813 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.686757    2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-bound-sa-token\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 17:59:03.686813 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.686766    2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab82597c-dd5c-45ee-86c7-51768a677041-registry-tls\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 17:59:03.686813 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.686775    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab82597c-dd5c-45ee-86c7-51768a677041-trusted-ca\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 17:59:03.686813 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.686782    2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab82597c-dd5c-45ee-86c7-51768a677041-installation-pull-secrets\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 17:59:03.686813 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.686791    2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab82597c-dd5c-45ee-86c7-51768a677041-image-registry-private-configuration\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 17:59:03.686813 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.686804    2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab82597c-dd5c-45ee-86c7-51768a677041-ca-trust-extracted\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 17:59:03.891042 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.891012    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65d8456986-md2f4"]
Sep 17 17:59:03.901318 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.901290    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65d8456986-md2f4"]
Sep 17 17:59:03.937647 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:03.937564    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab82597c-dd5c-45ee-86c7-51768a677041" path="/var/lib/kubelet/pods/ab82597c-dd5c-45ee-86c7-51768a677041/volumes"
Sep 17 17:59:04.208463 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:04.208383    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:04.306877 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:04.306842    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:04.411410 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:04.411371    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:05.127468 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:05.127433    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:05.262952 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:05.262864    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:05.352910 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:05.352876    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:07.009311 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.009263    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:07.108268 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.108236    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:07.796357 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.796322    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:07.898272 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.898230    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:07.930240 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.930213    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d9d67765c-kvgms"]
Sep 17 17:59:07.930586 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.930564    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab82597c-dd5c-45ee-86c7-51768a677041" containerName="registry"
Sep 17 17:59:07.930586 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.930581    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab82597c-dd5c-45ee-86c7-51768a677041" containerName="registry"
Sep 17 17:59:07.930761 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.930632    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab82597c-dd5c-45ee-86c7-51768a677041" containerName="registry"
Sep 17 17:59:07.935285 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.935266    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:07.943832 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.943786    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:59:07.943832 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.943820    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:59:07.944778 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.944759    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:59:07.944876 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.944787    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:59:07.944876 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.944801    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q5c6r\""
Sep 17 17:59:07.944876 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.944823    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:59:07.951652 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:07.951635    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d9d67765c-kvgms"]
Sep 17 17:59:08.021791 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.021769    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-oauth-config\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.022148 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.021804    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-oauth-serving-cert\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.022148 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.021825    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-serving-cert\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.022148 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.021839    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2spph\" (UniqueName: \"kubernetes.io/projected/be7fa4ca-8201-40d9-a205-ab3c22fb543f-kube-api-access-2spph\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.022148 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.021862    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-service-ca\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.022148 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.021918    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-config\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.122824 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.122750    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-config\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.122968 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.122833    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-oauth-config\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.122968 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.122874    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-oauth-serving-cert\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.122968 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.122900    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-serving-cert\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.123124 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.123034    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2spph\" (UniqueName: \"kubernetes.io/projected/be7fa4ca-8201-40d9-a205-ab3c22fb543f-kube-api-access-2spph\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.123124 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.123083    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-service-ca\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.123565 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.123534    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-oauth-serving-cert\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.123698 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.123678    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-config\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.123796 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.123776    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-service-ca\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.125273 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.125257    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-oauth-config\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.125324 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.125265    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-serving-cert\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.131441 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.131424    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2spph\" (UniqueName: \"kubernetes.io/projected/be7fa4ca-8201-40d9-a205-ab3c22fb543f-kube-api-access-2spph\") pod \"console-6d9d67765c-kvgms\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") " pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:08.245374 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:08.245343    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:09.794243 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:09.794207    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:09.907399 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:09.907367    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:12.398733 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:12.398694    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:12.471241 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:12.471198    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:15.133290 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:15.133210    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d9d67765c-kvgms"]
Sep 17 17:59:15.136717 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:59:15.136689    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7fa4ca_8201_40d9_a205_ab3c22fb543f.slice/crio-dd58fea67570ee9e4e6755af4a901121c4d7d02df8d54b51d31729597e2e25b0 WatchSource:0}: Error finding container dd58fea67570ee9e4e6755af4a901121c4d7d02df8d54b51d31729597e2e25b0: Status 404 returned error can't find the container with id dd58fea67570ee9e4e6755af4a901121c4d7d02df8d54b51d31729597e2e25b0
Sep 17 17:59:15.455223 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:15.455140    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:15.597341 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:15.597295    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:15.602186 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:15.602155    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-8qvts" event={"ID":"d79f9b27-81c4-47aa-b378-ed1d73cd2b03","Type":"ContainerStarted","Data":"37f0a4cdc6b135e7be6d33dc5bb3f89018b72c2c512523143a8cf60afa9b713d"}
Sep 17 17:59:15.602385 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:15.602367    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-8qvts"
Sep 17 17:59:15.604593 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:15.604569    2574 generic.go:356] "Generic (PLEG): container finished" podID="04d65cc5-2f68-45e9-8359-fff61bd12996" containerID="be81f90f065fd9b0628ea938b81a57140eaa645d30843cbbfd0c01edc4ccdb1d" exitCode=0
Sep 17 17:59:15.604696 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:15.604648    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-flfs2" event={"ID":"04d65cc5-2f68-45e9-8359-fff61bd12996","Type":"ContainerDied","Data":"be81f90f065fd9b0628ea938b81a57140eaa645d30843cbbfd0c01edc4ccdb1d"}
Sep 17 17:59:15.605000 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:15.604984    2574 scope.go:117] "RemoveContainer" containerID="be81f90f065fd9b0628ea938b81a57140eaa645d30843cbbfd0c01edc4ccdb1d"
Sep 17 17:59:15.606125 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:15.606103    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9d67765c-kvgms" event={"ID":"be7fa4ca-8201-40d9-a205-ab3c22fb543f","Type":"ContainerStarted","Data":"dd58fea67570ee9e4e6755af4a901121c4d7d02df8d54b51d31729597e2e25b0"}
Sep 17 17:59:15.614873 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:15.614854    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-8qvts"
Sep 17 17:59:15.670407 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:15.668741    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-8qvts" podStartSLOduration=1.636809676 podStartE2EDuration="17.668719119s" totalImagesPullingTime="16.031909443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:15.63631455 +0000 UTC m=+125.937136872" watchObservedRunningTime="2026-09-17 17:59:15.668719119 +0000 UTC m=+125.969541447"
Sep 17 17:59:16.615362 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:16.615321    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-flfs2" event={"ID":"04d65cc5-2f68-45e9-8359-fff61bd12996","Type":"ContainerStarted","Data":"758cf607a63dba4c39c6da4f1dcec9a306dd19aef82ad300c6f25caa07ff4a59"}
Sep 17 17:59:18.039627 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.039589    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fdd48dddf-2pxs5"]
Sep 17 17:59:18.060540 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.060475    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.074860 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.074686    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:59:18.077554 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.077472    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdd48dddf-2pxs5"]
Sep 17 17:59:18.215454 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.215418    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqf5s\" (UniqueName: \"kubernetes.io/projected/17b64ee2-2da0-4d2b-9673-2e04c1427c14-kube-api-access-vqf5s\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.215643 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.215470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-service-ca\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.215643 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.215550    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-oauth-serving-cert\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.215643 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.215581    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-config\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.215643 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.215600    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-trusted-ca-bundle\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.215643 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.215630    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-oauth-config\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.215873 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.215709    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-serving-cert\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.316659 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.316578    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-config\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.316659 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.316627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-trusted-ca-bundle\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.316875 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.316667    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-oauth-config\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.316875 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.316746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-serving-cert\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.316875 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.316780    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqf5s\" (UniqueName: \"kubernetes.io/projected/17b64ee2-2da0-4d2b-9673-2e04c1427c14-kube-api-access-vqf5s\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.316875 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.316815    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-service-ca\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.316875 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.316874    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-oauth-serving-cert\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.317666 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.317634    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-oauth-serving-cert\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.318240 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.318214    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-config\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.319059 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.319036    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-trusted-ca-bundle\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.320103 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.320051    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-service-ca\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.322051 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.321988    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-oauth-config\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.322498 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.322476    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-serving-cert\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.336695 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.336653    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqf5s\" (UniqueName: \"kubernetes.io/projected/17b64ee2-2da0-4d2b-9673-2e04c1427c14-kube-api-access-vqf5s\") pod \"console-5fdd48dddf-2pxs5\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") " pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.376712 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.376502    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:18.646142 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:18.646029    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-sm8vh_12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a/monitoring-plugin/0.log"
Sep 17 17:59:19.421150 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:19.421007    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:19.511170 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:19.511097    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:19.630445 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:19.630409    2574 generic.go:356] "Generic (PLEG): container finished" podID="8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5" containerID="48279015c93d61313a59a0fd58b2cec329f5277bc8b31ad921de76c7262db57a" exitCode=0
Sep 17 17:59:19.630668 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:19.630467    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6" event={"ID":"8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5","Type":"ContainerDied","Data":"48279015c93d61313a59a0fd58b2cec329f5277bc8b31ad921de76c7262db57a"}
Sep 17 17:59:19.630981 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:19.630958    2574 scope.go:117] "RemoveContainer" containerID="48279015c93d61313a59a0fd58b2cec329f5277bc8b31ad921de76c7262db57a"
Sep 17 17:59:20.046169 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:20.046126    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpx8b_0cecc0ad-f2e1-488b-a2f7-d46f669e8012/init-textfile/0.log"
Sep 17 17:59:20.252583 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:20.252557    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpx8b_0cecc0ad-f2e1-488b-a2f7-d46f669e8012/node-exporter/0.log"
Sep 17 17:59:20.447947 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:20.447872    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpx8b_0cecc0ad-f2e1-488b-a2f7-d46f669e8012/kube-rbac-proxy/0.log"
Sep 17 17:59:20.641614 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:20.641054    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs\") pod \"network-metrics-daemon-xjh98\" (UID: \"3ee045a7-cfc1-40b2-87a8-65bedec89f66\") " pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:59:20.644008 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:20.643985    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee045a7-cfc1-40b2-87a8-65bedec89f66-metrics-certs\") pod \"network-metrics-daemon-xjh98\" (UID: \"3ee045a7-cfc1-40b2-87a8-65bedec89f66\") " pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:59:20.745160 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:20.745077    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:20.747202 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:20.747180    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cf4sb\""
Sep 17 17:59:20.755613 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:20.755584    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xjh98"
Sep 17 17:59:20.829614 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:20.829586    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:20.843818 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:20.843788    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdd48dddf-2pxs5"]
Sep 17 17:59:20.925673 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:59:20.925641    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b64ee2_2da0_4d2b_9673_2e04c1427c14.slice/crio-9c4780b582afd62ad6e30469ce95e643d6773de0e3820af2c188237dd6af5389 WatchSource:0}: Error finding container 9c4780b582afd62ad6e30469ce95e643d6773de0e3820af2c188237dd6af5389: Status 404 returned error can't find the container with id 9c4780b582afd62ad6e30469ce95e643d6773de0e3820af2c188237dd6af5389
Sep 17 17:59:21.237836 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:21.237801    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xjh98"]
Sep 17 17:59:21.241425 ip-10-0-132-0 kubenswrapper[2574]: W0917 17:59:21.241397    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee045a7_cfc1_40b2_87a8_65bedec89f66.slice/crio-fc451c86329ecc80fecec90bfc1a554feaf754abbdb8dc1222a9f51b338e865f WatchSource:0}: Error finding container fc451c86329ecc80fecec90bfc1a554feaf754abbdb8dc1222a9f51b338e865f: Status 404 returned error can't find the container with id fc451c86329ecc80fecec90bfc1a554feaf754abbdb8dc1222a9f51b338e865f
Sep 17 17:59:21.641420 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:21.641384    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdd48dddf-2pxs5" event={"ID":"17b64ee2-2da0-4d2b-9673-2e04c1427c14","Type":"ContainerStarted","Data":"e0f4a946e9443cbcc25b73a89657f2b821232af5ee1342b11b9c327e583ef586"}
Sep 17 17:59:21.641923 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:21.641440    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdd48dddf-2pxs5" event={"ID":"17b64ee2-2da0-4d2b-9673-2e04c1427c14","Type":"ContainerStarted","Data":"9c4780b582afd62ad6e30469ce95e643d6773de0e3820af2c188237dd6af5389"}
Sep 17 17:59:21.643335 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:21.643290    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xjh98" event={"ID":"3ee045a7-cfc1-40b2-87a8-65bedec89f66","Type":"ContainerStarted","Data":"fc451c86329ecc80fecec90bfc1a554feaf754abbdb8dc1222a9f51b338e865f"}
Sep 17 17:59:21.645390 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:21.645366    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-4bbj6" event={"ID":"8cffa332-a0e2-4ec9-aaa4-db8efa6fdbb5","Type":"ContainerStarted","Data":"32fbb6b27cde29a832968404556881c06e330ed54f0c7739079da6459e308d37"}
Sep 17 17:59:21.647237 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:21.647211    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9d67765c-kvgms" event={"ID":"be7fa4ca-8201-40d9-a205-ab3c22fb543f","Type":"ContainerStarted","Data":"22a4bc68baf1504e484ca3b97787ee9d4c83bd3740d78b15b3abf835d2772159"}
Sep 17 17:59:21.740853 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:21.740783    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fdd48dddf-2pxs5" podStartSLOduration=3.7407582440000002 podStartE2EDuration="3.740758244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:21.731244177 +0000 UTC m=+132.032066521" watchObservedRunningTime="2026-09-17 17:59:21.740758244 +0000 UTC m=+132.041580568"
Sep 17 17:59:21.881181 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:21.881049    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d9d67765c-kvgms" podStartSLOduration=9.064119926 podStartE2EDuration="14.881029097s" totalImagesPullingTime="5.816909171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:21.879543147 +0000 UTC m=+132.180365471" watchObservedRunningTime="2026-09-17 17:59:21.881029097 +0000 UTC m=+132.181851420"
Sep 17 17:59:23.046415 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:23.046393    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-8l6qg_043673ea-e295-4b8a-a582-3af684b752b1/prometheus-operator-admission-webhook/0.log"
Sep 17 17:59:23.659363 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:23.659317    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xjh98" event={"ID":"3ee045a7-cfc1-40b2-87a8-65bedec89f66","Type":"ContainerStarted","Data":"e740be39ee5fa38ce90759cd832557f8c3c1c5fe2deee6a40ff60c5ef2f94125"}
Sep 17 17:59:23.659584 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:23.659393    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xjh98" event={"ID":"3ee045a7-cfc1-40b2-87a8-65bedec89f66","Type":"ContainerStarted","Data":"2d25356268ed39b115956de50a9d49ea571694be5327fc11de2ca4e57585cdba"}
Sep 17 17:59:23.689419 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:23.689364    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xjh98" podStartSLOduration=130.167599572 podStartE2EDuration="2m11.689340853s" totalImagesPullingTime="1.521741281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:23.688336973 +0000 UTC m=+133.989159297" watchObservedRunningTime="2026-09-17 17:59:23.689340853 +0000 UTC m=+133.990163176"
Sep 17 17:59:23.892430 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:23.892380    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:24.020965 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:24.020903    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:24.445730 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:24.445661    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-gp5gh_15845110-5c22-4b7c-b600-86ec6e146fe1/networking-console-plugin/0.log"
Sep 17 17:59:25.048599 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:25.048574    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d9d67765c-kvgms_be7fa4ca-8201-40d9-a205-ab3c22fb543f/console/0.log"
Sep 17 17:59:25.248079 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:25.248042    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-8qvts_d79f9b27-81c4-47aa-b378-ed1d73cd2b03/download-server/0.log"
Sep 17 17:59:26.667835 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:26.667790    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:26.804951 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:26.804920    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:28.049532 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:28.049478    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:28.134753 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:28.134721    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:28.245952 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:28.245921    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:28.246095 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:28.246010    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:28.251427 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:28.251405    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:28.377028 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:28.376955    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:28.377028 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:28.377020    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:28.382208 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:28.382185    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:28.685501 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:28.685427    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:28.685971 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:28.685949    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 17:59:28.862708 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:28.862665    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d9d67765c-kvgms"]
Sep 17 17:59:31.415257 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:31.415225    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:31.497099 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:31.497025    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:34.219166 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:34.219122    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:34.305940 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:34.305911    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:50.717423 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:50.717342    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:50.819901 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:50.819872    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:55.705294 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:55.705256    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d9d67765c-kvgms" podUID="be7fa4ca-8201-40d9-a205-ab3c22fb543f" containerName="console" containerID="cri-o://22a4bc68baf1504e484ca3b97787ee9d4c83bd3740d78b15b3abf835d2772159" gracePeriod=15
Sep 17 17:59:55.972037 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:55.971989    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:56.105219 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.105193    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-oauth-serving-cert\") pod \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") "
Sep 17 17:59:56.105374 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.105227    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-oauth-config\") pod \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") "
Sep 17 17:59:56.105374 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.105242    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-serving-cert\") pod \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") "
Sep 17 17:59:56.105374 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.105284    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2spph\" (UniqueName: \"kubernetes.io/projected/be7fa4ca-8201-40d9-a205-ab3c22fb543f-kube-api-access-2spph\") pod \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") "
Sep 17 17:59:56.105580 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.105494    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-config\") pod \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") "
Sep 17 17:59:56.105636 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.105586    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-service-ca\") pod \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\" (UID: \"be7fa4ca-8201-40d9-a205-ab3c22fb543f\") "
Sep 17 17:59:56.105636 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.105599    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "be7fa4ca-8201-40d9-a205-ab3c22fb543f" (UID: "be7fa4ca-8201-40d9-a205-ab3c22fb543f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:56.105828 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.105803    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-oauth-serving-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 17:59:56.105945 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.105799    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-config" (OuterVolumeSpecName: "console-config") pod "be7fa4ca-8201-40d9-a205-ab3c22fb543f" (UID: "be7fa4ca-8201-40d9-a205-ab3c22fb543f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:56.105997 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.105971    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-service-ca" (OuterVolumeSpecName: "service-ca") pod "be7fa4ca-8201-40d9-a205-ab3c22fb543f" (UID: "be7fa4ca-8201-40d9-a205-ab3c22fb543f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:56.107658 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.107635    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "be7fa4ca-8201-40d9-a205-ab3c22fb543f" (UID: "be7fa4ca-8201-40d9-a205-ab3c22fb543f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:56.107725 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.107693    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "be7fa4ca-8201-40d9-a205-ab3c22fb543f" (UID: "be7fa4ca-8201-40d9-a205-ab3c22fb543f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:56.107725 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.107711    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7fa4ca-8201-40d9-a205-ab3c22fb543f-kube-api-access-2spph" (OuterVolumeSpecName: "kube-api-access-2spph") pod "be7fa4ca-8201-40d9-a205-ab3c22fb543f" (UID: "be7fa4ca-8201-40d9-a205-ab3c22fb543f"). InnerVolumeSpecName "kube-api-access-2spph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:56.206304 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.206272    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2spph\" (UniqueName: \"kubernetes.io/projected/be7fa4ca-8201-40d9-a205-ab3c22fb543f-kube-api-access-2spph\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 17:59:56.206304 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.206303    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-config\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 17:59:56.206424 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.206312    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be7fa4ca-8201-40d9-a205-ab3c22fb543f-service-ca\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 17:59:56.206424 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.206320    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-oauth-config\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 17:59:56.206424 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.206329    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be7fa4ca-8201-40d9-a205-ab3c22fb543f-console-serving-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 17:59:56.778306 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.778267    2574 generic.go:356] "Generic (PLEG): container finished" podID="be7fa4ca-8201-40d9-a205-ab3c22fb543f" containerID="22a4bc68baf1504e484ca3b97787ee9d4c83bd3740d78b15b3abf835d2772159" exitCode=0
Sep 17 17:59:56.778701 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.778316    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9d67765c-kvgms"
Sep 17 17:59:56.778701 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.778324    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9d67765c-kvgms" event={"ID":"be7fa4ca-8201-40d9-a205-ab3c22fb543f","Type":"ContainerDied","Data":"22a4bc68baf1504e484ca3b97787ee9d4c83bd3740d78b15b3abf835d2772159"}
Sep 17 17:59:56.778701 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.778370    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9d67765c-kvgms" event={"ID":"be7fa4ca-8201-40d9-a205-ab3c22fb543f","Type":"ContainerDied","Data":"dd58fea67570ee9e4e6755af4a901121c4d7d02df8d54b51d31729597e2e25b0"}
Sep 17 17:59:56.778701 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.778391    2574 scope.go:117] "RemoveContainer" containerID="22a4bc68baf1504e484ca3b97787ee9d4c83bd3740d78b15b3abf835d2772159"
Sep 17 17:59:56.786348 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.786331    2574 scope.go:117] "RemoveContainer" containerID="22a4bc68baf1504e484ca3b97787ee9d4c83bd3740d78b15b3abf835d2772159"
Sep 17 17:59:56.786643 ip-10-0-132-0 kubenswrapper[2574]: E0917 17:59:56.786624    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a4bc68baf1504e484ca3b97787ee9d4c83bd3740d78b15b3abf835d2772159\": container with ID starting with 22a4bc68baf1504e484ca3b97787ee9d4c83bd3740d78b15b3abf835d2772159 not found: ID does not exist" containerID="22a4bc68baf1504e484ca3b97787ee9d4c83bd3740d78b15b3abf835d2772159"
Sep 17 17:59:56.786695 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.786651    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a4bc68baf1504e484ca3b97787ee9d4c83bd3740d78b15b3abf835d2772159"} err="failed to get container status \"22a4bc68baf1504e484ca3b97787ee9d4c83bd3740d78b15b3abf835d2772159\": rpc error: code = NotFound desc = could not find container \"22a4bc68baf1504e484ca3b97787ee9d4c83bd3740d78b15b3abf835d2772159\": container with ID starting with 22a4bc68baf1504e484ca3b97787ee9d4c83bd3740d78b15b3abf835d2772159 not found: ID does not exist"
Sep 17 17:59:56.804409 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.804391    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d9d67765c-kvgms"]
Sep 17 17:59:56.812839 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:56.812816    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d9d67765c-kvgms"]
Sep 17 17:59:57.935903 ip-10-0-132-0 kubenswrapper[2574]: I0917 17:59:57.935864    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be7fa4ca-8201-40d9-a205-ab3c22fb543f" path="/var/lib/kubelet/pods/be7fa4ca-8201-40d9-a205-ab3c22fb543f/volumes"
Sep 17 18:00:07.009355 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:07.009319    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:07.070972 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:07.070940    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:14.485469 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.485431    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d8679f985-2gz9j"]
Sep 17 18:00:14.485926 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.485817    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be7fa4ca-8201-40d9-a205-ab3c22fb543f" containerName="console"
Sep 17 18:00:14.485926 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.485838    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="be7fa4ca-8201-40d9-a205-ab3c22fb543f" containerName="console"
Sep 17 18:00:14.485926 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.485906    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="be7fa4ca-8201-40d9-a205-ab3c22fb543f" containerName="console"
Sep 17 18:00:14.488669 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.488649    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.500862 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.500842    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d8679f985-2gz9j"]
Sep 17 18:00:14.551202 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.551181    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:14.631692 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.631671    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:14.635675 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.635657    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-config\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.635728 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.635686    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-serving-cert\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.635728 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.635710    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-trusted-ca-bundle\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.635796 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.635779    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-oauth-config\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.635828 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.635808    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l524v\" (UniqueName: \"kubernetes.io/projected/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-kube-api-access-l524v\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.635828 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.635825    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-oauth-serving-cert\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.635886 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.635846    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-service-ca\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.736808 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.736761    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-config\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.736808 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.736787    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-serving-cert\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.736938 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.736815    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-trusted-ca-bundle\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.736938 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.736838    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-oauth-config\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.737014 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.736948    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l524v\" (UniqueName: \"kubernetes.io/projected/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-kube-api-access-l524v\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.737014 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.736976    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-oauth-serving-cert\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.737014 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.736998    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-service-ca\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.737614 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.737594    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-config\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.737676 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.737628    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-service-ca\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.737676 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.737653    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-oauth-serving-cert\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.737676 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.737669    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-trusted-ca-bundle\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.739900 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.739881    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-oauth-config\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.740100 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.740080    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-serving-cert\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.746021 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.745999    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l524v\" (UniqueName: \"kubernetes.io/projected/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-kube-api-access-l524v\") pod \"console-6d8679f985-2gz9j\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") " pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:14.796901 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:14.796867    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:15.074993 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:15.074967    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d8679f985-2gz9j"]
Sep 17 18:00:15.076438 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:00:15.076412    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7dcf8c_fde6_4a38_ba9f_487d4a06a27c.slice/crio-62083a6fa2c8ab0bc1597ef2b2d396ca64b7b74dc3f833afbfb9ca08a00b0fcf WatchSource:0}: Error finding container 62083a6fa2c8ab0bc1597ef2b2d396ca64b7b74dc3f833afbfb9ca08a00b0fcf: Status 404 returned error can't find the container with id 62083a6fa2c8ab0bc1597ef2b2d396ca64b7b74dc3f833afbfb9ca08a00b0fcf
Sep 17 18:00:15.845661 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:15.845629    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8679f985-2gz9j" event={"ID":"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c","Type":"ContainerStarted","Data":"b8068b24ca8bc785f018fc45aa33e3a5565fb58b43d76bd25031e9b473415c05"}
Sep 17 18:00:15.846018 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:15.845676    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8679f985-2gz9j" event={"ID":"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c","Type":"ContainerStarted","Data":"62083a6fa2c8ab0bc1597ef2b2d396ca64b7b74dc3f833afbfb9ca08a00b0fcf"}
Sep 17 18:00:15.867123 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:15.867080    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d8679f985-2gz9j" podStartSLOduration=1.867065564 podStartE2EDuration="1.867065564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:15.865791832 +0000 UTC m=+186.166614178" watchObservedRunningTime="2026-09-17 18:00:15.867065564 +0000 UTC m=+186.167887887"
Sep 17 18:00:24.797483 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:24.797441    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:24.797978 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:24.797533    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:24.802307 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:24.802282    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:24.880405 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:24.880386    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:00:24.929621 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:24.929592    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fdd48dddf-2pxs5"]
Sep 17 18:00:25.052452 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:25.052380    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:25.152662 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:25.152634    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:25.226254 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:25.226230    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:46.728802 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:46.728769    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:46.796215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:46.796181    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:49.948538 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:49.948478    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fdd48dddf-2pxs5" podUID="17b64ee2-2da0-4d2b-9673-2e04c1427c14" containerName="console" containerID="cri-o://e0f4a946e9443cbcc25b73a89657f2b821232af5ee1342b11b9c327e583ef586" gracePeriod=15
Sep 17 18:00:50.186586 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.186562    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 18:00:50.382776 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.382740    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-serving-cert\") pod \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") "
Sep 17 18:00:50.382776 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.382779    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-oauth-config\") pod \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") "
Sep 17 18:00:50.383013 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.382805    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-service-ca\") pod \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") "
Sep 17 18:00:50.383013 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.382837    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-trusted-ca-bundle\") pod \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") "
Sep 17 18:00:50.383013 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.382878    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-oauth-serving-cert\") pod \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") "
Sep 17 18:00:50.383013 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.382928    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqf5s\" (UniqueName: \"kubernetes.io/projected/17b64ee2-2da0-4d2b-9673-2e04c1427c14-kube-api-access-vqf5s\") pod \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") "
Sep 17 18:00:50.383013 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.382954    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-config\") pod \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\" (UID: \"17b64ee2-2da0-4d2b-9673-2e04c1427c14\") "
Sep 17 18:00:50.383364 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.383254    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-service-ca" (OuterVolumeSpecName: "service-ca") pod "17b64ee2-2da0-4d2b-9673-2e04c1427c14" (UID: "17b64ee2-2da0-4d2b-9673-2e04c1427c14"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:50.383364 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.383272    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "17b64ee2-2da0-4d2b-9673-2e04c1427c14" (UID: "17b64ee2-2da0-4d2b-9673-2e04c1427c14"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:50.383486 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.383359    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "17b64ee2-2da0-4d2b-9673-2e04c1427c14" (UID: "17b64ee2-2da0-4d2b-9673-2e04c1427c14"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:50.383486 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.383437    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-config" (OuterVolumeSpecName: "console-config") pod "17b64ee2-2da0-4d2b-9673-2e04c1427c14" (UID: "17b64ee2-2da0-4d2b-9673-2e04c1427c14"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:50.385068 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.385042    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "17b64ee2-2da0-4d2b-9673-2e04c1427c14" (UID: "17b64ee2-2da0-4d2b-9673-2e04c1427c14"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:50.385642 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.385618    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "17b64ee2-2da0-4d2b-9673-2e04c1427c14" (UID: "17b64ee2-2da0-4d2b-9673-2e04c1427c14"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:50.385709 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.385620    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b64ee2-2da0-4d2b-9673-2e04c1427c14-kube-api-access-vqf5s" (OuterVolumeSpecName: "kube-api-access-vqf5s") pod "17b64ee2-2da0-4d2b-9673-2e04c1427c14" (UID: "17b64ee2-2da0-4d2b-9673-2e04c1427c14"). InnerVolumeSpecName "kube-api-access-vqf5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:50.484284 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.484254    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-trusted-ca-bundle\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:00:50.484284 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.484280    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-oauth-serving-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:00:50.484442 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.484293    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqf5s\" (UniqueName: \"kubernetes.io/projected/17b64ee2-2da0-4d2b-9673-2e04c1427c14-kube-api-access-vqf5s\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:00:50.484442 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.484306    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-config\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:00:50.484442 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.484318    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-serving-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:00:50.484442 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.484329    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17b64ee2-2da0-4d2b-9673-2e04c1427c14-console-oauth-config\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:00:50.484442 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.484339    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17b64ee2-2da0-4d2b-9673-2e04c1427c14-service-ca\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:00:50.965002 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.964968    2574 generic.go:356] "Generic (PLEG): container finished" podID="17b64ee2-2da0-4d2b-9673-2e04c1427c14" containerID="e0f4a946e9443cbcc25b73a89657f2b821232af5ee1342b11b9c327e583ef586" exitCode=0
Sep 17 18:00:50.965455 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.965010    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdd48dddf-2pxs5" event={"ID":"17b64ee2-2da0-4d2b-9673-2e04c1427c14","Type":"ContainerDied","Data":"e0f4a946e9443cbcc25b73a89657f2b821232af5ee1342b11b9c327e583ef586"}
Sep 17 18:00:50.965455 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.965022    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdd48dddf-2pxs5"
Sep 17 18:00:50.965455 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.965040    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdd48dddf-2pxs5" event={"ID":"17b64ee2-2da0-4d2b-9673-2e04c1427c14","Type":"ContainerDied","Data":"9c4780b582afd62ad6e30469ce95e643d6773de0e3820af2c188237dd6af5389"}
Sep 17 18:00:50.965455 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.965058    2574 scope.go:117] "RemoveContainer" containerID="e0f4a946e9443cbcc25b73a89657f2b821232af5ee1342b11b9c327e583ef586"
Sep 17 18:00:50.974161 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.974143    2574 scope.go:117] "RemoveContainer" containerID="e0f4a946e9443cbcc25b73a89657f2b821232af5ee1342b11b9c327e583ef586"
Sep 17 18:00:50.974428 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:00:50.974409    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f4a946e9443cbcc25b73a89657f2b821232af5ee1342b11b9c327e583ef586\": container with ID starting with e0f4a946e9443cbcc25b73a89657f2b821232af5ee1342b11b9c327e583ef586 not found: ID does not exist" containerID="e0f4a946e9443cbcc25b73a89657f2b821232af5ee1342b11b9c327e583ef586"
Sep 17 18:00:50.974486 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.974439    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f4a946e9443cbcc25b73a89657f2b821232af5ee1342b11b9c327e583ef586"} err="failed to get container status \"e0f4a946e9443cbcc25b73a89657f2b821232af5ee1342b11b9c327e583ef586\": rpc error: code = NotFound desc = could not find container \"e0f4a946e9443cbcc25b73a89657f2b821232af5ee1342b11b9c327e583ef586\": container with ID starting with e0f4a946e9443cbcc25b73a89657f2b821232af5ee1342b11b9c327e583ef586 not found: ID does not exist"
Sep 17 18:00:50.987072 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.987050    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fdd48dddf-2pxs5"]
Sep 17 18:00:50.991444 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:50.991419    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fdd48dddf-2pxs5"]
Sep 17 18:00:51.936813 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:00:51.936783    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17b64ee2-2da0-4d2b-9673-2e04c1427c14" path="/var/lib/kubelet/pods/17b64ee2-2da0-4d2b-9673-2e04c1427c14/volumes"
Sep 17 18:01:22.024503 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:22.024413    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:22.101546 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:22.101497    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:34.041471 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:34.041434    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:34.125936 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:34.125900    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:53.449442 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:53.449408    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:53.555741 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:53.555703    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:58.357103 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.357070    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-845cc4bd59-llwmb"]
Sep 17 18:01:58.357444 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.357318    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b64ee2-2da0-4d2b-9673-2e04c1427c14" containerName="console"
Sep 17 18:01:58.357444 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.357328    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b64ee2-2da0-4d2b-9673-2e04c1427c14" containerName="console"
Sep 17 18:01:58.357444 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.357381    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b64ee2-2da0-4d2b-9673-2e04c1427c14" containerName="console"
Sep 17 18:01:58.359137 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.359122    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.430506 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.430478    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:58.443522 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.443497    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e92cb037-a5c6-4139-949d-ffd44119896d-console-oauth-config\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.443601 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.443549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-oauth-serving-cert\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.443601 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.443574    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-trusted-ca-bundle\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.443687 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.443599    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-service-ca\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.443687 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.443632    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e92cb037-a5c6-4139-949d-ffd44119896d-console-serving-cert\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.443687 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.443666    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tjn4\" (UniqueName: \"kubernetes.io/projected/e92cb037-a5c6-4139-949d-ffd44119896d-kube-api-access-9tjn4\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.443687 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.443683    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-console-config\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.447475 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.447452    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-845cc4bd59-llwmb"]
Sep 17 18:01:58.536004 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.535956    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:58.544053 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.544030    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-trusted-ca-bundle\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.544153 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.544056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-service-ca\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.544153 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.544076    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e92cb037-a5c6-4139-949d-ffd44119896d-console-serving-cert\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.544153 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.544112    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjn4\" (UniqueName: \"kubernetes.io/projected/e92cb037-a5c6-4139-949d-ffd44119896d-kube-api-access-9tjn4\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.544313 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.544273    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-console-config\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.544367 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.544327    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e92cb037-a5c6-4139-949d-ffd44119896d-console-oauth-config\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.544419 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.544373    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-oauth-serving-cert\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.544853 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.544829    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-service-ca\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.544967 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.544943    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-console-config\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.545059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.545039    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-trusted-ca-bundle\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.545104 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.545086    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-oauth-serving-cert\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.546771 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.546749    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e92cb037-a5c6-4139-949d-ffd44119896d-console-serving-cert\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.546855 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.546820    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e92cb037-a5c6-4139-949d-ffd44119896d-console-oauth-config\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.566671 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.566655    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjn4\" (UniqueName: \"kubernetes.io/projected/e92cb037-a5c6-4139-949d-ffd44119896d-kube-api-access-9tjn4\") pod \"console-845cc4bd59-llwmb\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") " pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.667838 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.667778    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:01:58.965067 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:58.964999    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-845cc4bd59-llwmb"]
Sep 17 18:01:58.968266 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:01:58.968236    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92cb037_a5c6_4139_949d_ffd44119896d.slice/crio-c456b1b020d39f8d6730a3ce51250125e1092ab61b4fcd3fe73d0171b62ba963 WatchSource:0}: Error finding container c456b1b020d39f8d6730a3ce51250125e1092ab61b4fcd3fe73d0171b62ba963: Status 404 returned error can't find the container with id c456b1b020d39f8d6730a3ce51250125e1092ab61b4fcd3fe73d0171b62ba963
Sep 17 18:01:59.208394 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:59.208355    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845cc4bd59-llwmb" event={"ID":"e92cb037-a5c6-4139-949d-ffd44119896d","Type":"ContainerStarted","Data":"09c42cc0920ce439091e942d4b1b72b6260670c41d7f0e588e8a0f27809cc40f"}
Sep 17 18:01:59.208550 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:59.208409    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845cc4bd59-llwmb" event={"ID":"e92cb037-a5c6-4139-949d-ffd44119896d","Type":"ContainerStarted","Data":"c456b1b020d39f8d6730a3ce51250125e1092ab61b4fcd3fe73d0171b62ba963"}
Sep 17 18:01:59.244003 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:01:59.243927    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-845cc4bd59-llwmb" podStartSLOduration=1.243912445 podStartE2EDuration="1.243912445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:59.243573969 +0000 UTC m=+289.544396294" watchObservedRunningTime="2026-09-17 18:01:59.243912445 +0000 UTC m=+289.544734768"
Sep 17 18:02:02.125299 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:02.125264    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:03.532299 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:03.532265    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:03.646898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:03.646867    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:08.668382 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:08.668329    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:02:08.668382 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:08.668397    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:02:08.673175 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:08.673152    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:02:09.248631 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:09.248603    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:02:09.306449 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:09.306414    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d8679f985-2gz9j"]
Sep 17 18:02:09.449300 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:09.449271    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:09.548310 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:09.548279    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:09.615070 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:09.615039    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:11.737943 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:11.737916    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:02:11.741824 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:11.741803    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:02:11.744397 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:11.744378    2574 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:02:22.919197 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:22.919160    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:23.794184 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:23.794154    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:23.894523 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:23.894490    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:23.993684 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:23.993655    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:24.019035 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.019010    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"]
Sep 17 18:02:24.021190 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.021176    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"
Sep 17 18:02:24.023392 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.023370    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 18:02:24.024315 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.024296    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 18:02:24.024555 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.024538    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 18:02:24.024604 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.024540    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 18:02:24.032917 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.032899    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"]
Sep 17 18:02:24.136557 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.136457    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c342133f-27ec-4c59-8e61-154d464ea090-klusterlet-config\") pod \"klusterlet-addon-workmgr-558b7858cf-76n4r\" (UID: \"c342133f-27ec-4c59-8e61-154d464ea090\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"
Sep 17 18:02:24.136557 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.136494    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c342133f-27ec-4c59-8e61-154d464ea090-tmp\") pod \"klusterlet-addon-workmgr-558b7858cf-76n4r\" (UID: \"c342133f-27ec-4c59-8e61-154d464ea090\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"
Sep 17 18:02:24.136735 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.136587    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7qkp\" (UniqueName: \"kubernetes.io/projected/c342133f-27ec-4c59-8e61-154d464ea090-kube-api-access-f7qkp\") pod \"klusterlet-addon-workmgr-558b7858cf-76n4r\" (UID: \"c342133f-27ec-4c59-8e61-154d464ea090\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"
Sep 17 18:02:24.236908 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.236879    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7qkp\" (UniqueName: \"kubernetes.io/projected/c342133f-27ec-4c59-8e61-154d464ea090-kube-api-access-f7qkp\") pod \"klusterlet-addon-workmgr-558b7858cf-76n4r\" (UID: \"c342133f-27ec-4c59-8e61-154d464ea090\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"
Sep 17 18:02:24.237075 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.236951    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c342133f-27ec-4c59-8e61-154d464ea090-klusterlet-config\") pod \"klusterlet-addon-workmgr-558b7858cf-76n4r\" (UID: \"c342133f-27ec-4c59-8e61-154d464ea090\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"
Sep 17 18:02:24.237075 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.236980    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c342133f-27ec-4c59-8e61-154d464ea090-tmp\") pod \"klusterlet-addon-workmgr-558b7858cf-76n4r\" (UID: \"c342133f-27ec-4c59-8e61-154d464ea090\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"
Sep 17 18:02:24.237340 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.237320    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c342133f-27ec-4c59-8e61-154d464ea090-tmp\") pod \"klusterlet-addon-workmgr-558b7858cf-76n4r\" (UID: \"c342133f-27ec-4c59-8e61-154d464ea090\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"
Sep 17 18:02:24.239621 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.239605    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c342133f-27ec-4c59-8e61-154d464ea090-klusterlet-config\") pod \"klusterlet-addon-workmgr-558b7858cf-76n4r\" (UID: \"c342133f-27ec-4c59-8e61-154d464ea090\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"
Sep 17 18:02:24.244815 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.244794    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7qkp\" (UniqueName: \"kubernetes.io/projected/c342133f-27ec-4c59-8e61-154d464ea090-kube-api-access-f7qkp\") pod \"klusterlet-addon-workmgr-558b7858cf-76n4r\" (UID: \"c342133f-27ec-4c59-8e61-154d464ea090\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"
Sep 17 18:02:24.329924 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.329902    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"
Sep 17 18:02:24.599370 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.599339    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"]
Sep 17 18:02:24.602500 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:02:24.602477    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc342133f_27ec_4c59_8e61_154d464ea090.slice/crio-d15dcda9cad19de550eca8b5cd138f1f095ec103f501ffa250056d2ce7cc35d3 WatchSource:0}: Error finding container d15dcda9cad19de550eca8b5cd138f1f095ec103f501ffa250056d2ce7cc35d3: Status 404 returned error can't find the container with id d15dcda9cad19de550eca8b5cd138f1f095ec103f501ffa250056d2ce7cc35d3
Sep 17 18:02:24.604252 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:24.604234    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:02:25.305808 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:25.305770    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r" event={"ID":"c342133f-27ec-4c59-8e61-154d464ea090","Type":"ContainerStarted","Data":"d15dcda9cad19de550eca8b5cd138f1f095ec103f501ffa250056d2ce7cc35d3"}
Sep 17 18:02:28.322306 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:28.322270    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r" event={"ID":"c342133f-27ec-4c59-8e61-154d464ea090","Type":"ContainerStarted","Data":"a0589e632af940c2ac43fb43624f4dbae03e7aec0c813de8c3a315b6bcf968aa"}
Sep 17 18:02:28.322746 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:28.322485    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"
Sep 17 18:02:28.323797 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:28.323777    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r"
Sep 17 18:02:28.339505 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:28.339454    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-558b7858cf-76n4r" podStartSLOduration=0.72306964 podStartE2EDuration="4.33943939s" totalImagesPullingTime="3.61636975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:28.338262129 +0000 UTC m=+318.639084455" watchObservedRunningTime="2026-09-17 18:02:28.33943939 +0000 UTC m=+318.640261713"
Sep 17 18:02:34.326450 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.326405    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d8679f985-2gz9j" podUID="7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c" containerName="console" containerID="cri-o://b8068b24ca8bc785f018fc45aa33e3a5565fb58b43d76bd25031e9b473415c05" gracePeriod=15
Sep 17 18:02:34.565559 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.565536    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:02:34.720660 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.720601    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-oauth-serving-cert\") pod \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") "
Sep 17 18:02:34.720660 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.720638    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-service-ca\") pod \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") "
Sep 17 18:02:34.720827 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.720670    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-trusted-ca-bundle\") pod \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") "
Sep 17 18:02:34.720827 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.720732    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-config\") pod \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") "
Sep 17 18:02:34.720827 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.720760    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-serving-cert\") pod \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") "
Sep 17 18:02:34.720827 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.720784    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-oauth-config\") pod \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") "
Sep 17 18:02:34.720827 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.720809    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l524v\" (UniqueName: \"kubernetes.io/projected/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-kube-api-access-l524v\") pod \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\" (UID: \"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c\") "
Sep 17 18:02:34.721063 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.720941    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c" (UID: "7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:34.721123 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.721071    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-service-ca" (OuterVolumeSpecName: "service-ca") pod "7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c" (UID: "7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:34.721199 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.721171    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-config" (OuterVolumeSpecName: "console-config") pod "7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c" (UID: "7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:34.721254 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.721205    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c" (UID: "7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:34.721337 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.721321    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-oauth-serving-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:02:34.721393 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.721345    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-service-ca\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:02:34.721393 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.721359    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-trusted-ca-bundle\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:02:34.721393 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.721373    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-config\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:02:34.723067 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.723038    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c" (UID: "7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:34.723156 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.723082    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c" (UID: "7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:34.723156 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.723117    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-kube-api-access-l524v" (OuterVolumeSpecName: "kube-api-access-l524v") pod "7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c" (UID: "7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c"). InnerVolumeSpecName "kube-api-access-l524v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:34.822064 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.822043    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-serving-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:02:34.822064 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.822062    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-console-oauth-config\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:02:34.822197 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:34.822070    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l524v\" (UniqueName: \"kubernetes.io/projected/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c-kube-api-access-l524v\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:02:35.350358 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:35.350328    2574 generic.go:356] "Generic (PLEG): container finished" podID="7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c" containerID="b8068b24ca8bc785f018fc45aa33e3a5565fb58b43d76bd25031e9b473415c05" exitCode=0
Sep 17 18:02:35.350732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:35.350370    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d8679f985-2gz9j"
Sep 17 18:02:35.350732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:35.350411    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8679f985-2gz9j" event={"ID":"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c","Type":"ContainerDied","Data":"b8068b24ca8bc785f018fc45aa33e3a5565fb58b43d76bd25031e9b473415c05"}
Sep 17 18:02:35.350732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:35.350459    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d8679f985-2gz9j" event={"ID":"7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c","Type":"ContainerDied","Data":"62083a6fa2c8ab0bc1597ef2b2d396ca64b7b74dc3f833afbfb9ca08a00b0fcf"}
Sep 17 18:02:35.350732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:35.350478    2574 scope.go:117] "RemoveContainer" containerID="b8068b24ca8bc785f018fc45aa33e3a5565fb58b43d76bd25031e9b473415c05"
Sep 17 18:02:35.360584 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:35.360567    2574 scope.go:117] "RemoveContainer" containerID="b8068b24ca8bc785f018fc45aa33e3a5565fb58b43d76bd25031e9b473415c05"
Sep 17 18:02:35.360829 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:02:35.360810    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8068b24ca8bc785f018fc45aa33e3a5565fb58b43d76bd25031e9b473415c05\": container with ID starting with b8068b24ca8bc785f018fc45aa33e3a5565fb58b43d76bd25031e9b473415c05 not found: ID does not exist" containerID="b8068b24ca8bc785f018fc45aa33e3a5565fb58b43d76bd25031e9b473415c05"
Sep 17 18:02:35.360883 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:35.360840    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8068b24ca8bc785f018fc45aa33e3a5565fb58b43d76bd25031e9b473415c05"} err="failed to get container status \"b8068b24ca8bc785f018fc45aa33e3a5565fb58b43d76bd25031e9b473415c05\": rpc error: code = NotFound desc = could not find container \"b8068b24ca8bc785f018fc45aa33e3a5565fb58b43d76bd25031e9b473415c05\": container with ID starting with b8068b24ca8bc785f018fc45aa33e3a5565fb58b43d76bd25031e9b473415c05 not found: ID does not exist"
Sep 17 18:02:35.373344 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:35.373318    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d8679f985-2gz9j"]
Sep 17 18:02:35.376992 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:35.376973    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d8679f985-2gz9j"]
Sep 17 18:02:35.936500 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:02:35.936470    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c" path="/var/lib/kubelet/pods/7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c/volumes"
Sep 17 18:03:31.030160 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:31.030128    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:32.158440 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.158403    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:32.233102 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.233067    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"]
Sep 17 18:03:32.233340 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.233328    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c" containerName="console"
Sep 17 18:03:32.233382 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.233341    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c" containerName="console"
Sep 17 18:03:32.233416 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.233393    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7dcf8c-fde6-4a38-ba9f-487d4a06a27c" containerName="console"
Sep 17 18:03:32.236650 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.236618    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"
Sep 17 18:03:32.239934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.239915    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lc8mp\""
Sep 17 18:03:32.240409 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.240390    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:03:32.240826 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.240811    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:03:32.247944 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.247923    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"]
Sep 17 18:03:32.318759 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.318725    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e88b3ad1-af48-4310-ad17-59064b36b696-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd\" (UID: \"e88b3ad1-af48-4310-ad17-59064b36b696\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"
Sep 17 18:03:32.318945 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.318781    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e88b3ad1-af48-4310-ad17-59064b36b696-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd\" (UID: \"e88b3ad1-af48-4310-ad17-59064b36b696\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"
Sep 17 18:03:32.318945 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.318834    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpdn\" (UniqueName: \"kubernetes.io/projected/e88b3ad1-af48-4310-ad17-59064b36b696-kube-api-access-hnpdn\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd\" (UID: \"e88b3ad1-af48-4310-ad17-59064b36b696\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"
Sep 17 18:03:32.420119 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.420030    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e88b3ad1-af48-4310-ad17-59064b36b696-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd\" (UID: \"e88b3ad1-af48-4310-ad17-59064b36b696\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"
Sep 17 18:03:32.420119 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.420090    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e88b3ad1-af48-4310-ad17-59064b36b696-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd\" (UID: \"e88b3ad1-af48-4310-ad17-59064b36b696\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"
Sep 17 18:03:32.420119 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.420116    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpdn\" (UniqueName: \"kubernetes.io/projected/e88b3ad1-af48-4310-ad17-59064b36b696-kube-api-access-hnpdn\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd\" (UID: \"e88b3ad1-af48-4310-ad17-59064b36b696\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"
Sep 17 18:03:32.420440 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.420420    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e88b3ad1-af48-4310-ad17-59064b36b696-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd\" (UID: \"e88b3ad1-af48-4310-ad17-59064b36b696\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"
Sep 17 18:03:32.420484 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.420451    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e88b3ad1-af48-4310-ad17-59064b36b696-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd\" (UID: \"e88b3ad1-af48-4310-ad17-59064b36b696\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"
Sep 17 18:03:32.429569 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.429539    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpdn\" (UniqueName: \"kubernetes.io/projected/e88b3ad1-af48-4310-ad17-59064b36b696-kube-api-access-hnpdn\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd\" (UID: \"e88b3ad1-af48-4310-ad17-59064b36b696\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"
Sep 17 18:03:32.547673 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.547633    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"
Sep 17 18:03:32.819370 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:32.819336    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"]
Sep 17 18:03:32.822247 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:03:32.822210    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88b3ad1_af48_4310_ad17_59064b36b696.slice/crio-52c6dffbf947892d7306ce14bd30ef6ded81db18bb05165b63ef3aef623f1f41 WatchSource:0}: Error finding container 52c6dffbf947892d7306ce14bd30ef6ded81db18bb05165b63ef3aef623f1f41: Status 404 returned error can't find the container with id 52c6dffbf947892d7306ce14bd30ef6ded81db18bb05165b63ef3aef623f1f41
Sep 17 18:03:33.561638 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:33.561600    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd" event={"ID":"e88b3ad1-af48-4310-ad17-59064b36b696","Type":"ContainerStarted","Data":"52c6dffbf947892d7306ce14bd30ef6ded81db18bb05165b63ef3aef623f1f41"}
Sep 17 18:03:39.589976 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:39.589941    2574 generic.go:356] "Generic (PLEG): container finished" podID="e88b3ad1-af48-4310-ad17-59064b36b696" containerID="ade9792b5a2d9e3aa369fccb11cfb89ac57c8a690d5c091afef312619c0adca3" exitCode=0
Sep 17 18:03:39.590434 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:39.590023    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd" event={"ID":"e88b3ad1-af48-4310-ad17-59064b36b696","Type":"ContainerDied","Data":"ade9792b5a2d9e3aa369fccb11cfb89ac57c8a690d5c091afef312619c0adca3"}
Sep 17 18:03:41.603461 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:41.603431    2574 generic.go:356] "Generic (PLEG): container finished" podID="e88b3ad1-af48-4310-ad17-59064b36b696" containerID="560f338ed2da587553e3b061d29525a74bc8db3a617896bfae155faf154a9b78" exitCode=0
Sep 17 18:03:41.603806 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:41.603509    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd" event={"ID":"e88b3ad1-af48-4310-ad17-59064b36b696","Type":"ContainerDied","Data":"560f338ed2da587553e3b061d29525a74bc8db3a617896bfae155faf154a9b78"}
Sep 17 18:03:49.637403 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:49.637367    2574 generic.go:356] "Generic (PLEG): container finished" podID="e88b3ad1-af48-4310-ad17-59064b36b696" containerID="718ade0c69446509394f9005cf940f1b7655576be1979bc1388cd9c8c46bdefe" exitCode=0
Sep 17 18:03:49.637809 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:49.637449    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd" event={"ID":"e88b3ad1-af48-4310-ad17-59064b36b696","Type":"ContainerDied","Data":"718ade0c69446509394f9005cf940f1b7655576be1979bc1388cd9c8c46bdefe"}
Sep 17 18:03:50.762427 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:50.762401    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"
Sep 17 18:03:50.875800 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:50.875775    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnpdn\" (UniqueName: \"kubernetes.io/projected/e88b3ad1-af48-4310-ad17-59064b36b696-kube-api-access-hnpdn\") pod \"e88b3ad1-af48-4310-ad17-59064b36b696\" (UID: \"e88b3ad1-af48-4310-ad17-59064b36b696\") "
Sep 17 18:03:50.875959 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:50.875813    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e88b3ad1-af48-4310-ad17-59064b36b696-bundle\") pod \"e88b3ad1-af48-4310-ad17-59064b36b696\" (UID: \"e88b3ad1-af48-4310-ad17-59064b36b696\") "
Sep 17 18:03:50.875959 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:50.875848    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e88b3ad1-af48-4310-ad17-59064b36b696-util\") pod \"e88b3ad1-af48-4310-ad17-59064b36b696\" (UID: \"e88b3ad1-af48-4310-ad17-59064b36b696\") "
Sep 17 18:03:50.876459 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:50.876433    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88b3ad1-af48-4310-ad17-59064b36b696-bundle" (OuterVolumeSpecName: "bundle") pod "e88b3ad1-af48-4310-ad17-59064b36b696" (UID: "e88b3ad1-af48-4310-ad17-59064b36b696"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:03:50.878163 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:50.878130    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88b3ad1-af48-4310-ad17-59064b36b696-kube-api-access-hnpdn" (OuterVolumeSpecName: "kube-api-access-hnpdn") pod "e88b3ad1-af48-4310-ad17-59064b36b696" (UID: "e88b3ad1-af48-4310-ad17-59064b36b696"). InnerVolumeSpecName "kube-api-access-hnpdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:03:50.880899 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:50.880877    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88b3ad1-af48-4310-ad17-59064b36b696-util" (OuterVolumeSpecName: "util") pod "e88b3ad1-af48-4310-ad17-59064b36b696" (UID: "e88b3ad1-af48-4310-ad17-59064b36b696"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:03:50.977118 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:50.977054    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnpdn\" (UniqueName: \"kubernetes.io/projected/e88b3ad1-af48-4310-ad17-59064b36b696-kube-api-access-hnpdn\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:03:50.977118 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:50.977080    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e88b3ad1-af48-4310-ad17-59064b36b696-bundle\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:03:50.977118 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:50.977089    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e88b3ad1-af48-4310-ad17-59064b36b696-util\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:03:51.651456 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:51.651421    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd" event={"ID":"e88b3ad1-af48-4310-ad17-59064b36b696","Type":"ContainerDied","Data":"52c6dffbf947892d7306ce14bd30ef6ded81db18bb05165b63ef3aef623f1f41"}
Sep 17 18:03:51.651456 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:51.651458    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1aglltd"
Sep 17 18:03:51.651658 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:51.651494    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52c6dffbf947892d7306ce14bd30ef6ded81db18bb05165b63ef3aef623f1f41"
Sep 17 18:03:53.996732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:53.996699    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:54.113254 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:54.113215    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:55.855295 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:55.855250    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2"]
Sep 17 18:03:55.855866 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:55.855842    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e88b3ad1-af48-4310-ad17-59064b36b696" containerName="extract"
Sep 17 18:03:55.855929 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:55.855870    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e88b3ad1-af48-4310-ad17-59064b36b696" containerName="extract"
Sep 17 18:03:55.855929 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:55.855885    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e88b3ad1-af48-4310-ad17-59064b36b696" containerName="pull"
Sep 17 18:03:55.855929 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:55.855894    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e88b3ad1-af48-4310-ad17-59064b36b696" containerName="pull"
Sep 17 18:03:55.855929 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:55.855927    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e88b3ad1-af48-4310-ad17-59064b36b696" containerName="util"
Sep 17 18:03:55.856095 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:55.855941    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e88b3ad1-af48-4310-ad17-59064b36b696" containerName="util"
Sep 17 18:03:55.856095 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:55.856066    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e88b3ad1-af48-4310-ad17-59064b36b696" containerName="extract"
Sep 17 18:03:55.860921 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:55.860898    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2"
Sep 17 18:03:55.866191 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:55.866171    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 18:03:55.867305 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:55.867084    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 18:03:55.867305 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:55.867100    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 18:03:55.867963 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:55.867948    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-qn5tk\""
Sep 17 18:03:55.916455 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:55.916429    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2"]
Sep 17 18:03:56.010367 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:56.010331    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlln2\" (UniqueName: \"kubernetes.io/projected/c17f6dd0-274d-458d-ac16-a86c0d44aa35-kube-api-access-rlln2\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-pdcg2\" (UID: \"c17f6dd0-274d-458d-ac16-a86c0d44aa35\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2"
Sep 17 18:03:56.010558 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:56.010387    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c17f6dd0-274d-458d-ac16-a86c0d44aa35-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-pdcg2\" (UID: \"c17f6dd0-274d-458d-ac16-a86c0d44aa35\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2"
Sep 17 18:03:56.110991 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:56.110902    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlln2\" (UniqueName: \"kubernetes.io/projected/c17f6dd0-274d-458d-ac16-a86c0d44aa35-kube-api-access-rlln2\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-pdcg2\" (UID: \"c17f6dd0-274d-458d-ac16-a86c0d44aa35\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2"
Sep 17 18:03:56.110991 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:56.110968    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c17f6dd0-274d-458d-ac16-a86c0d44aa35-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-pdcg2\" (UID: \"c17f6dd0-274d-458d-ac16-a86c0d44aa35\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2"
Sep 17 18:03:56.113753 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:56.113719    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c17f6dd0-274d-458d-ac16-a86c0d44aa35-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-pdcg2\" (UID: \"c17f6dd0-274d-458d-ac16-a86c0d44aa35\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2"
Sep 17 18:03:56.120073 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:56.120049    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlln2\" (UniqueName: \"kubernetes.io/projected/c17f6dd0-274d-458d-ac16-a86c0d44aa35-kube-api-access-rlln2\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-pdcg2\" (UID: \"c17f6dd0-274d-458d-ac16-a86c0d44aa35\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2"
Sep 17 18:03:56.171076 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:56.171041    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2"
Sep 17 18:03:56.442410 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:56.442333    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2"]
Sep 17 18:03:56.445811 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:03:56.445782    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17f6dd0_274d_458d_ac16_a86c0d44aa35.slice/crio-5961c348b19d406a2d5aebce0b4c7435f15b4709417ea34e2dd07acbbf2eb5d7 WatchSource:0}: Error finding container 5961c348b19d406a2d5aebce0b4c7435f15b4709417ea34e2dd07acbbf2eb5d7: Status 404 returned error can't find the container with id 5961c348b19d406a2d5aebce0b4c7435f15b4709417ea34e2dd07acbbf2eb5d7
Sep 17 18:03:56.670872 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:56.670840    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2" event={"ID":"c17f6dd0-274d-458d-ac16-a86c0d44aa35","Type":"ContainerStarted","Data":"5961c348b19d406a2d5aebce0b4c7435f15b4709417ea34e2dd07acbbf2eb5d7"}
Sep 17 18:03:59.687296 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:59.687261    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2" event={"ID":"c17f6dd0-274d-458d-ac16-a86c0d44aa35","Type":"ContainerStarted","Data":"efeff2c3fd19af5312bbd63c054bdb295dd8079d91162e606761944e21634cb4"}
Sep 17 18:03:59.687700 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:59.687410    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2"
Sep 17 18:03:59.713770 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:59.713719    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2" podStartSLOduration=1.6620578670000001 podStartE2EDuration="4.713702268s" totalImagesPullingTime="3.051644401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:59.712215856 +0000 UTC m=+410.013038202" watchObservedRunningTime="2026-09-17 18:03:59.713702268 +0000 UTC m=+410.014524592"
Sep 17 18:03:59.998305 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:03:59.998269    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:00.270168 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.270136    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-hgw9w"]
Sep 17 18:04:00.272553 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.272534    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:00.274695 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.274672    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 18:04:00.274928 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.274904    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 18:04:00.275102 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.275085    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-7kdsx\""
Sep 17 18:04:00.285644 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.285616    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-hgw9w"]
Sep 17 18:04:00.285760 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.285669    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:00.347183 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.347150    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8x69\" (UniqueName: \"kubernetes.io/projected/a378b7de-100c-4fdb-991e-fef6f01acfb9-kube-api-access-k8x69\") pod \"keda-operator-579c8958f7-hgw9w\" (UID: \"a378b7de-100c-4fdb-991e-fef6f01acfb9\") " pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:00.347354 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.347193    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a378b7de-100c-4fdb-991e-fef6f01acfb9-cabundle0\") pod \"keda-operator-579c8958f7-hgw9w\" (UID: \"a378b7de-100c-4fdb-991e-fef6f01acfb9\") " pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:00.347354 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.347220    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a378b7de-100c-4fdb-991e-fef6f01acfb9-certificates\") pod \"keda-operator-579c8958f7-hgw9w\" (UID: \"a378b7de-100c-4fdb-991e-fef6f01acfb9\") " pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:00.447745 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.447710    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8x69\" (UniqueName: \"kubernetes.io/projected/a378b7de-100c-4fdb-991e-fef6f01acfb9-kube-api-access-k8x69\") pod \"keda-operator-579c8958f7-hgw9w\" (UID: \"a378b7de-100c-4fdb-991e-fef6f01acfb9\") " pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:00.447933 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.447772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a378b7de-100c-4fdb-991e-fef6f01acfb9-cabundle0\") pod \"keda-operator-579c8958f7-hgw9w\" (UID: \"a378b7de-100c-4fdb-991e-fef6f01acfb9\") " pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:00.447933 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.447827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a378b7de-100c-4fdb-991e-fef6f01acfb9-certificates\") pod \"keda-operator-579c8958f7-hgw9w\" (UID: \"a378b7de-100c-4fdb-991e-fef6f01acfb9\") " pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:00.448084 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:04:00.447968    2574 secret.go:281] references non-existent secret key: ca.crt
Sep 17 18:04:00.448084 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:04:00.447985    2574 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 18:04:00.448084 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:04:00.447997    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-hgw9w: references non-existent secret key: ca.crt
Sep 17 18:04:00.448084 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:04:00.448052    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a378b7de-100c-4fdb-991e-fef6f01acfb9-certificates podName:a378b7de-100c-4fdb-991e-fef6f01acfb9 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:00.948030884 +0000 UTC m=+411.248853189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a378b7de-100c-4fdb-991e-fef6f01acfb9-certificates") pod "keda-operator-579c8958f7-hgw9w" (UID: "a378b7de-100c-4fdb-991e-fef6f01acfb9") : references non-existent secret key: ca.crt
Sep 17 18:04:00.448566 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.448541    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a378b7de-100c-4fdb-991e-fef6f01acfb9-cabundle0\") pod \"keda-operator-579c8958f7-hgw9w\" (UID: \"a378b7de-100c-4fdb-991e-fef6f01acfb9\") " pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:00.461248 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.461218    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8x69\" (UniqueName: \"kubernetes.io/projected/a378b7de-100c-4fdb-991e-fef6f01acfb9-kube-api-access-k8x69\") pod \"keda-operator-579c8958f7-hgw9w\" (UID: \"a378b7de-100c-4fdb-991e-fef6f01acfb9\") " pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:00.517897 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.517828    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:00.627530 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.627422    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:00.725675 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.725643    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"]
Sep 17 18:04:00.728116 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.728101    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"
Sep 17 18:04:00.731669 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.731649    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 18:04:00.750069 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.750041    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"]
Sep 17 18:04:00.849863 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.849826    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47r55\" (UniqueName: \"kubernetes.io/projected/a87d48de-9b9c-49ff-a667-dba7f3525e67-kube-api-access-47r55\") pod \"keda-metrics-apiserver-77857b7b98-kh25n\" (UID: \"a87d48de-9b9c-49ff-a667-dba7f3525e67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"
Sep 17 18:04:00.850032 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.849949    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a87d48de-9b9c-49ff-a667-dba7f3525e67-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-kh25n\" (UID: \"a87d48de-9b9c-49ff-a667-dba7f3525e67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"
Sep 17 18:04:00.850032 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.850000    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a87d48de-9b9c-49ff-a667-dba7f3525e67-certificates\") pod \"keda-metrics-apiserver-77857b7b98-kh25n\" (UID: \"a87d48de-9b9c-49ff-a667-dba7f3525e67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"
Sep 17 18:04:00.929696 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.929599    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:00.950915 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.950884    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a378b7de-100c-4fdb-991e-fef6f01acfb9-certificates\") pod \"keda-operator-579c8958f7-hgw9w\" (UID: \"a378b7de-100c-4fdb-991e-fef6f01acfb9\") " pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:00.951006 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.950931    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a87d48de-9b9c-49ff-a667-dba7f3525e67-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-kh25n\" (UID: \"a87d48de-9b9c-49ff-a667-dba7f3525e67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"
Sep 17 18:04:00.951006 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.950953    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a87d48de-9b9c-49ff-a667-dba7f3525e67-certificates\") pod \"keda-metrics-apiserver-77857b7b98-kh25n\" (UID: \"a87d48de-9b9c-49ff-a667-dba7f3525e67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"
Sep 17 18:04:00.951006 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.950985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47r55\" (UniqueName: \"kubernetes.io/projected/a87d48de-9b9c-49ff-a667-dba7f3525e67-kube-api-access-47r55\") pod \"keda-metrics-apiserver-77857b7b98-kh25n\" (UID: \"a87d48de-9b9c-49ff-a667-dba7f3525e67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"
Sep 17 18:04:00.951116 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:04:00.951035    2574 secret.go:281] references non-existent secret key: ca.crt
Sep 17 18:04:00.951116 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:04:00.951054    2574 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 18:04:00.951116 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:04:00.951066    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-hgw9w: references non-existent secret key: ca.crt
Sep 17 18:04:00.951214 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:04:00.951125    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a378b7de-100c-4fdb-991e-fef6f01acfb9-certificates podName:a378b7de-100c-4fdb-991e-fef6f01acfb9 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:01.951107866 +0000 UTC m=+412.251930170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a378b7de-100c-4fdb-991e-fef6f01acfb9-certificates") pod "keda-operator-579c8958f7-hgw9w" (UID: "a378b7de-100c-4fdb-991e-fef6f01acfb9") : references non-existent secret key: ca.crt
Sep 17 18:04:00.951214 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:04:00.951131    2574 secret.go:281] references non-existent secret key: tls.crt
Sep 17 18:04:00.951214 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:04:00.951152    2574 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 17 18:04:00.951214 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:04:00.951176    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n: references non-existent secret key: tls.crt
Sep 17 18:04:00.951354 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:04:00.951241    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a87d48de-9b9c-49ff-a667-dba7f3525e67-certificates podName:a87d48de-9b9c-49ff-a667-dba7f3525e67 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:01.451224965 +0000 UTC m=+411.752047283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a87d48de-9b9c-49ff-a667-dba7f3525e67-certificates") pod "keda-metrics-apiserver-77857b7b98-kh25n" (UID: "a87d48de-9b9c-49ff-a667-dba7f3525e67") : references non-existent secret key: tls.crt
Sep 17 18:04:00.951396 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.951353    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a87d48de-9b9c-49ff-a667-dba7f3525e67-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-kh25n\" (UID: \"a87d48de-9b9c-49ff-a667-dba7f3525e67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"
Sep 17 18:04:00.959828 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:00.959802    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47r55\" (UniqueName: \"kubernetes.io/projected/a87d48de-9b9c-49ff-a667-dba7f3525e67-kube-api-access-47r55\") pod \"keda-metrics-apiserver-77857b7b98-kh25n\" (UID: \"a87d48de-9b9c-49ff-a667-dba7f3525e67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"
Sep 17 18:04:01.345215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.345178    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:01.409894 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.409855    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-xwwqb"]
Sep 17 18:04:01.412405 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.412386    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-xwwqb"
Sep 17 18:04:01.420442 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.420420    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 18:04:01.420739 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:04:01.420717    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"keda-admission-6c4d864b86-xwwqb\" is forbidden: User \"system:node:ip-10-0-132-0.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-keda\": no relationship found between node 'ip-10-0-132-0.ec2.internal' and this object" podUID="ff613b53-2fc9-4bef-960e-918d63f753fe" pod="openshift-keda/keda-admission-6c4d864b86-xwwqb"
Sep 17 18:04:01.435333 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.435308    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-xwwqb"]
Sep 17 18:04:01.455302 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.455278    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a87d48de-9b9c-49ff-a667-dba7f3525e67-certificates\") pod \"keda-metrics-apiserver-77857b7b98-kh25n\" (UID: \"a87d48de-9b9c-49ff-a667-dba7f3525e67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"
Sep 17 18:04:01.457960 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.457933    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a87d48de-9b9c-49ff-a667-dba7f3525e67-certificates\") pod \"keda-metrics-apiserver-77857b7b98-kh25n\" (UID: \"a87d48de-9b9c-49ff-a667-dba7f3525e67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"
Sep 17 18:04:01.491667 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.491635    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:01.556490 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.556458    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ff613b53-2fc9-4bef-960e-918d63f753fe-certificates\") pod \"keda-admission-6c4d864b86-xwwqb\" (UID: \"ff613b53-2fc9-4bef-960e-918d63f753fe\") " pod="openshift-keda/keda-admission-6c4d864b86-xwwqb"
Sep 17 18:04:01.556665 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.556500    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hskp7\" (UniqueName: \"kubernetes.io/projected/ff613b53-2fc9-4bef-960e-918d63f753fe-kube-api-access-hskp7\") pod \"keda-admission-6c4d864b86-xwwqb\" (UID: \"ff613b53-2fc9-4bef-960e-918d63f753fe\") " pod="openshift-keda/keda-admission-6c4d864b86-xwwqb"
Sep 17 18:04:01.593052 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.592998    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:01.637439 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.637342    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"
Sep 17 18:04:01.657909 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.657881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ff613b53-2fc9-4bef-960e-918d63f753fe-certificates\") pod \"keda-admission-6c4d864b86-xwwqb\" (UID: \"ff613b53-2fc9-4bef-960e-918d63f753fe\") " pod="openshift-keda/keda-admission-6c4d864b86-xwwqb"
Sep 17 18:04:01.658027 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.657927    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hskp7\" (UniqueName: \"kubernetes.io/projected/ff613b53-2fc9-4bef-960e-918d63f753fe-kube-api-access-hskp7\") pod \"keda-admission-6c4d864b86-xwwqb\" (UID: \"ff613b53-2fc9-4bef-960e-918d63f753fe\") " pod="openshift-keda/keda-admission-6c4d864b86-xwwqb"
Sep 17 18:04:01.660610 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.660591    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ff613b53-2fc9-4bef-960e-918d63f753fe-certificates\") pod \"keda-admission-6c4d864b86-xwwqb\" (UID: \"ff613b53-2fc9-4bef-960e-918d63f753fe\") " pod="openshift-keda/keda-admission-6c4d864b86-xwwqb"
Sep 17 18:04:01.668139 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.668112    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hskp7\" (UniqueName: \"kubernetes.io/projected/ff613b53-2fc9-4bef-960e-918d63f753fe-kube-api-access-hskp7\") pod \"keda-admission-6c4d864b86-xwwqb\" (UID: \"ff613b53-2fc9-4bef-960e-918d63f753fe\") " pod="openshift-keda/keda-admission-6c4d864b86-xwwqb"
Sep 17 18:04:01.721940 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.721888    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-xwwqb"
Sep 17 18:04:01.797501 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.797459    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:01.900268 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.900184    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:01.915291 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.915261    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"]
Sep 17 18:04:01.918962 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:04:01.918937    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87d48de_9b9c_49ff_a667_dba7f3525e67.slice/crio-76628f5363a0daa060b902e12f97d8f53dd86e0576ccb369b09d2569fad06fe8 WatchSource:0}: Error finding container 76628f5363a0daa060b902e12f97d8f53dd86e0576ccb369b09d2569fad06fe8: Status 404 returned error can't find the container with id 76628f5363a0daa060b902e12f97d8f53dd86e0576ccb369b09d2569fad06fe8
Sep 17 18:04:01.961782 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.961746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a378b7de-100c-4fdb-991e-fef6f01acfb9-certificates\") pod \"keda-operator-579c8958f7-hgw9w\" (UID: \"a378b7de-100c-4fdb-991e-fef6f01acfb9\") " pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:01.964814 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:01.964785    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a378b7de-100c-4fdb-991e-fef6f01acfb9-certificates\") pod \"keda-operator-579c8958f7-hgw9w\" (UID: \"a378b7de-100c-4fdb-991e-fef6f01acfb9\") " pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:02.004529 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:02.004485    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-xwwqb"]
Sep 17 18:04:02.006757 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:04:02.006729    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff613b53_2fc9_4bef_960e_918d63f753fe.slice/crio-61313344b7b53483feb8157afd913c716b875933fb23ffcc41885b701ad88428 WatchSource:0}: Error finding container 61313344b7b53483feb8157afd913c716b875933fb23ffcc41885b701ad88428: Status 404 returned error can't find the container with id 61313344b7b53483feb8157afd913c716b875933fb23ffcc41885b701ad88428
Sep 17 18:04:02.024098 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:02.024072    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:02.082589 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:02.082540    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:02.180197 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:02.180122    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:02.280718 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:02.280685    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:02.355923 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:02.355897    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-hgw9w"]
Sep 17 18:04:02.358463 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:04:02.358438    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda378b7de_100c_4fdb_991e_fef6f01acfb9.slice/crio-d78ccffda80a35bb0a1c7df532672a40f473874805d8e44c9bafd7100fed8bd2 WatchSource:0}: Error finding container d78ccffda80a35bb0a1c7df532672a40f473874805d8e44c9bafd7100fed8bd2: Status 404 returned error can't find the container with id d78ccffda80a35bb0a1c7df532672a40f473874805d8e44c9bafd7100fed8bd2
Sep 17 18:04:02.714256 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:02.714220    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-hgw9w" event={"ID":"a378b7de-100c-4fdb-991e-fef6f01acfb9","Type":"ContainerStarted","Data":"d78ccffda80a35bb0a1c7df532672a40f473874805d8e44c9bafd7100fed8bd2"}
Sep 17 18:04:02.715400 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:02.715375    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n" event={"ID":"a87d48de-9b9c-49ff-a667-dba7f3525e67","Type":"ContainerStarted","Data":"76628f5363a0daa060b902e12f97d8f53dd86e0576ccb369b09d2569fad06fe8"}
Sep 17 18:04:02.716556 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:02.716537    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-xwwqb" event={"ID":"ff613b53-2fc9-4bef-960e-918d63f753fe","Type":"ContainerStarted","Data":"61313344b7b53483feb8157afd913c716b875933fb23ffcc41885b701ad88428"}
Sep 17 18:04:04.727676 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:04.727638    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-xwwqb" event={"ID":"ff613b53-2fc9-4bef-960e-918d63f753fe","Type":"ContainerStarted","Data":"76d169e21f886f4e938ef135f620cd345d32fd7100f814ff71b4d9a15d3b1b63"}
Sep 17 18:04:04.728143 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:04.727745    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-xwwqb"
Sep 17 18:04:04.745709 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:04.745642    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-xwwqb" podStartSLOduration=1.744409543 podStartE2EDuration="3.74562416s" totalImagesPullingTime="2.001214617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:04.744813407 +0000 UTC m=+415.045635732" watchObservedRunningTime="2026-09-17 18:04:04.74562416 +0000 UTC m=+415.046446483"
Sep 17 18:04:07.744015 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:07.743978    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-hgw9w" event={"ID":"a378b7de-100c-4fdb-991e-fef6f01acfb9","Type":"ContainerStarted","Data":"5647b488cf7fed66b21b102a027fea723de7cf5b2341d2f8fbe5558b7d9a49a6"}
Sep 17 18:04:07.744397 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:07.744066    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:07.745852 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:07.745824    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n" event={"ID":"a87d48de-9b9c-49ff-a667-dba7f3525e67","Type":"ContainerStarted","Data":"8acd63619d27cca578d022959fb336a77e071841cc2eb5e92d444a9227659195"}
Sep 17 18:04:07.745965 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:07.745945    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"
Sep 17 18:04:07.809179 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:07.809128    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-hgw9w" podStartSLOduration=3.102211324 podStartE2EDuration="7.809108409s" totalImagesPullingTime="4.706897085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:07.770950316 +0000 UTC m=+418.071772677" watchObservedRunningTime="2026-09-17 18:04:07.809108409 +0000 UTC m=+418.109930732"
Sep 17 18:04:07.809561 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:07.809526    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n" podStartSLOduration=2.667198151 podStartE2EDuration="7.809496488s" totalImagesPullingTime="5.142298337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:07.80474965 +0000 UTC m=+418.105571988" watchObservedRunningTime="2026-09-17 18:04:07.809496488 +0000 UTC m=+418.110318814"
Sep 17 18:04:18.756444 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:18.756410    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kh25n"
Sep 17 18:04:20.693657 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:20.693582    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pdcg2"
Sep 17 18:04:22.234381 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.234350    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cb4977c64-r99sq"]
Sep 17 18:04:22.247155 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.247135    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.249781 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.249759    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cb4977c64-r99sq"]
Sep 17 18:04:22.315366 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.315337    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:22.328709 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.328682    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-service-ca\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.328836 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.328712    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvrz7\" (UniqueName: \"kubernetes.io/projected/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-kube-api-access-lvrz7\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.328836 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.328734    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-trusted-ca-bundle\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.328836 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.328820    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-console-serving-cert\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.328962 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.328862    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-console-config\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.328962 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.328893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-oauth-serving-cert\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.328962 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.328922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-console-oauth-config\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.415753 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.415723    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:22.429605 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.429582    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-console-config\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.429741 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.429632    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-oauth-serving-cert\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.429741 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.429675    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-console-oauth-config\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.429741 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.429725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-service-ca\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.429899 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.429746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvrz7\" (UniqueName: \"kubernetes.io/projected/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-kube-api-access-lvrz7\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.429899 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.429870    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-trusted-ca-bundle\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.429995 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.429925    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-console-serving-cert\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.430425 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.430402    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-oauth-serving-cert\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.430425 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.430402    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-console-config\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.430778 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.430607    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-service-ca\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.430778 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.430686    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-trusted-ca-bundle\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.432372 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.432347    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-console-oauth-config\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.432881 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.432860    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-console-serving-cert\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.439087 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.439058    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvrz7\" (UniqueName: \"kubernetes.io/projected/23b7236a-7fd4-44d2-bdcd-0faa73cf4e28-kube-api-access-lvrz7\") pod \"console-cb4977c64-r99sq\" (UID: \"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28\") " pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.490090 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.490034    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:22.559082 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.559060    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:22.857261 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:22.857231    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cb4977c64-r99sq"]
Sep 17 18:04:22.859195 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:04:22.859156    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b7236a_7fd4_44d2_bdcd_0faa73cf4e28.slice/crio-ce7790fb9dc424a9d033a8a37611414f22f6d69f4a5ee5058ff3d14a1497e976 WatchSource:0}: Error finding container ce7790fb9dc424a9d033a8a37611414f22f6d69f4a5ee5058ff3d14a1497e976: Status 404 returned error can't find the container with id ce7790fb9dc424a9d033a8a37611414f22f6d69f4a5ee5058ff3d14a1497e976
Sep 17 18:04:23.043501 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:23.043462    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:23.147233 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:23.147156    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:23.820177 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:23.820137    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb4977c64-r99sq" event={"ID":"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28","Type":"ContainerStarted","Data":"9f0b3e0a63b3b33e81ef2b96f47b29299fb3904a44b33001d8c3c4d4e2b30f45"}
Sep 17 18:04:23.820572 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:23.820204    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb4977c64-r99sq" event={"ID":"23b7236a-7fd4-44d2-bdcd-0faa73cf4e28","Type":"ContainerStarted","Data":"ce7790fb9dc424a9d033a8a37611414f22f6d69f4a5ee5058ff3d14a1497e976"}
Sep 17 18:04:23.845585 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:23.843774    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cb4977c64-r99sq" podStartSLOduration=1.843749811 podStartE2EDuration="1.843749811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:23.841851028 +0000 UTC m=+434.142673375" watchObservedRunningTime="2026-09-17 18:04:23.843749811 +0000 UTC m=+434.144572136"
Sep 17 18:04:25.735104 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:25.735068    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-xwwqb"
Sep 17 18:04:28.753301 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:28.753275    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-hgw9w"
Sep 17 18:04:32.560263 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:32.560217    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:32.560703 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:32.560291    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:32.565210 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:32.565189    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:32.869803 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:32.869727    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cb4977c64-r99sq"
Sep 17 18:04:32.919106 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:32.919074    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-845cc4bd59-llwmb"]
Sep 17 18:04:33.049447 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:33.049411    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:33.150994 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:33.150907    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:33.232171 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:33.232146    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:52.399322 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.399286    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:52.497451 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.497424    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:52.671253 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.671196    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"]
Sep 17 18:04:52.673711 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.673697    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"
Sep 17 18:04:52.676558 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.676539    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lc8mp\""
Sep 17 18:04:52.677049 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.677033    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:04:52.677530 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.677498    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:04:52.685097 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.685078    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"]
Sep 17 18:04:52.778090 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.778065    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g\" (UID: \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"
Sep 17 18:04:52.778189 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.778126    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9ln9\" (UniqueName: \"kubernetes.io/projected/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-kube-api-access-z9ln9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g\" (UID: \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"
Sep 17 18:04:52.778253 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.778232    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g\" (UID: \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"
Sep 17 18:04:52.879504 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.879477    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9ln9\" (UniqueName: \"kubernetes.io/projected/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-kube-api-access-z9ln9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g\" (UID: \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"
Sep 17 18:04:52.879644 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.879588    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g\" (UID: \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"
Sep 17 18:04:52.879644 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.879625    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g\" (UID: \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"
Sep 17 18:04:52.879956 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.879937    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g\" (UID: \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"
Sep 17 18:04:52.880027 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.879972    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g\" (UID: \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"
Sep 17 18:04:52.888422 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.888404    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9ln9\" (UniqueName: \"kubernetes.io/projected/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-kube-api-access-z9ln9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g\" (UID: \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"
Sep 17 18:04:52.982945 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:52.982892    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"
Sep 17 18:04:53.252841 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:53.252800    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"]
Sep 17 18:04:53.255202 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:04:53.255174    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0f4f9b_4db2_45cb_ac8e_a4d4deb30b67.slice/crio-3ca059c5f0568dcfeb7adbccbb9538602f14f604685485e25144d296b373bd3d WatchSource:0}: Error finding container 3ca059c5f0568dcfeb7adbccbb9538602f14f604685485e25144d296b373bd3d: Status 404 returned error can't find the container with id 3ca059c5f0568dcfeb7adbccbb9538602f14f604685485e25144d296b373bd3d
Sep 17 18:04:53.958022 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:53.957992    2574 generic.go:356] "Generic (PLEG): container finished" podID="0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67" containerID="193e06c9b8ed3d9ab32e5646d9917c4ef3e2f5327639575f8d8bdcc92053706e" exitCode=0
Sep 17 18:04:53.958401 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:53.958074    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g" event={"ID":"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67","Type":"ContainerDied","Data":"193e06c9b8ed3d9ab32e5646d9917c4ef3e2f5327639575f8d8bdcc92053706e"}
Sep 17 18:04:53.958401 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:53.958110    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g" event={"ID":"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67","Type":"ContainerStarted","Data":"3ca059c5f0568dcfeb7adbccbb9538602f14f604685485e25144d296b373bd3d"}
Sep 17 18:04:55.969142 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:55.969120    2574 generic.go:356] "Generic (PLEG): container finished" podID="0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67" containerID="563c5d57c995723801139cb3e6edf258f2b03ace4a141c74686439e8bfe9f0c6" exitCode=0
Sep 17 18:04:55.969457 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:55.969163    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g" event={"ID":"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67","Type":"ContainerDied","Data":"563c5d57c995723801139cb3e6edf258f2b03ace4a141c74686439e8bfe9f0c6"}
Sep 17 18:04:56.975803 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:56.975766    2574 generic.go:356] "Generic (PLEG): container finished" podID="0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67" containerID="aa62e5b876cee600a3b60fd106b8c78455480d11257f8a6eeedb7cf910a4cbdf" exitCode=0
Sep 17 18:04:56.976145 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:56.975836    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g" event={"ID":"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67","Type":"ContainerDied","Data":"aa62e5b876cee600a3b60fd106b8c78455480d11257f8a6eeedb7cf910a4cbdf"}
Sep 17 18:04:57.938423 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:57.938387    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-845cc4bd59-llwmb" podUID="e92cb037-a5c6-4139-949d-ffd44119896d" containerName="console" containerID="cri-o://09c42cc0920ce439091e942d4b1b72b6260670c41d7f0e588e8a0f27809cc40f" gracePeriod=15
Sep 17 18:04:58.106988 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.106964    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"
Sep 17 18:04:58.186879 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.186859    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:04:58.218740 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.218678    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-bundle\") pod \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\" (UID: \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\") "
Sep 17 18:04:58.218840 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.218770    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9ln9\" (UniqueName: \"kubernetes.io/projected/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-kube-api-access-z9ln9\") pod \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\" (UID: \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\") "
Sep 17 18:04:58.218840 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.218801    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-util\") pod \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\" (UID: \"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67\") "
Sep 17 18:04:58.219404 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.219378    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-bundle" (OuterVolumeSpecName: "bundle") pod "0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67" (UID: "0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:58.220896 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.220869    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-kube-api-access-z9ln9" (OuterVolumeSpecName: "kube-api-access-z9ln9") pod "0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67" (UID: "0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67"). InnerVolumeSpecName "kube-api-access-z9ln9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:58.228759 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.228723    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-util" (OuterVolumeSpecName: "util") pod "0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67" (UID: "0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:58.319506 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.319480    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-trusted-ca-bundle\") pod \"e92cb037-a5c6-4139-949d-ffd44119896d\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") "
Sep 17 18:04:58.319610 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.319529    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-oauth-serving-cert\") pod \"e92cb037-a5c6-4139-949d-ffd44119896d\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") "
Sep 17 18:04:58.319610 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.319548    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-service-ca\") pod \"e92cb037-a5c6-4139-949d-ffd44119896d\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") "
Sep 17 18:04:58.319610 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.319585    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tjn4\" (UniqueName: \"kubernetes.io/projected/e92cb037-a5c6-4139-949d-ffd44119896d-kube-api-access-9tjn4\") pod \"e92cb037-a5c6-4139-949d-ffd44119896d\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") "
Sep 17 18:04:58.319752 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.319619    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-console-config\") pod \"e92cb037-a5c6-4139-949d-ffd44119896d\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") "
Sep 17 18:04:58.319752 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.319636    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e92cb037-a5c6-4139-949d-ffd44119896d-console-oauth-config\") pod \"e92cb037-a5c6-4139-949d-ffd44119896d\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") "
Sep 17 18:04:58.319752 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.319650    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e92cb037-a5c6-4139-949d-ffd44119896d-console-serving-cert\") pod \"e92cb037-a5c6-4139-949d-ffd44119896d\" (UID: \"e92cb037-a5c6-4139-949d-ffd44119896d\") "
Sep 17 18:04:58.319890 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.319825    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9ln9\" (UniqueName: \"kubernetes.io/projected/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-kube-api-access-z9ln9\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:04:58.319890 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.319843    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-util\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:04:58.319890 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.319856    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67-bundle\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:04:58.320043 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.319967    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e92cb037-a5c6-4139-949d-ffd44119896d" (UID: "e92cb037-a5c6-4139-949d-ffd44119896d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:58.320043 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.319974    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e92cb037-a5c6-4139-949d-ffd44119896d" (UID: "e92cb037-a5c6-4139-949d-ffd44119896d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:58.320111 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.320040    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-service-ca" (OuterVolumeSpecName: "service-ca") pod "e92cb037-a5c6-4139-949d-ffd44119896d" (UID: "e92cb037-a5c6-4139-949d-ffd44119896d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:58.320111 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.320063    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-console-config" (OuterVolumeSpecName: "console-config") pod "e92cb037-a5c6-4139-949d-ffd44119896d" (UID: "e92cb037-a5c6-4139-949d-ffd44119896d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:58.321726 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.321696    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92cb037-a5c6-4139-949d-ffd44119896d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e92cb037-a5c6-4139-949d-ffd44119896d" (UID: "e92cb037-a5c6-4139-949d-ffd44119896d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:58.321873 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.321852    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92cb037-a5c6-4139-949d-ffd44119896d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e92cb037-a5c6-4139-949d-ffd44119896d" (UID: "e92cb037-a5c6-4139-949d-ffd44119896d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:58.321873 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.321862    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92cb037-a5c6-4139-949d-ffd44119896d-kube-api-access-9tjn4" (OuterVolumeSpecName: "kube-api-access-9tjn4") pod "e92cb037-a5c6-4139-949d-ffd44119896d" (UID: "e92cb037-a5c6-4139-949d-ffd44119896d"). InnerVolumeSpecName "kube-api-access-9tjn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:58.420519 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.420496    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-service-ca\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:04:58.420603 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.420535    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tjn4\" (UniqueName: \"kubernetes.io/projected/e92cb037-a5c6-4139-949d-ffd44119896d-kube-api-access-9tjn4\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:04:58.420603 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.420545    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-console-config\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:04:58.420603 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.420553    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e92cb037-a5c6-4139-949d-ffd44119896d-console-oauth-config\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:04:58.420603 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.420561    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e92cb037-a5c6-4139-949d-ffd44119896d-console-serving-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:04:58.420603 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.420568    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-trusted-ca-bundle\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:04:58.420603 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.420576    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e92cb037-a5c6-4139-949d-ffd44119896d-oauth-serving-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:04:58.985760 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.985730    2574 generic.go:356] "Generic (PLEG): container finished" podID="e92cb037-a5c6-4139-949d-ffd44119896d" containerID="09c42cc0920ce439091e942d4b1b72b6260670c41d7f0e588e8a0f27809cc40f" exitCode=0
Sep 17 18:04:58.985901 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.985773    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845cc4bd59-llwmb"
Sep 17 18:04:58.985901 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.985814    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845cc4bd59-llwmb" event={"ID":"e92cb037-a5c6-4139-949d-ffd44119896d","Type":"ContainerDied","Data":"09c42cc0920ce439091e942d4b1b72b6260670c41d7f0e588e8a0f27809cc40f"}
Sep 17 18:04:58.985901 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.985864    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845cc4bd59-llwmb" event={"ID":"e92cb037-a5c6-4139-949d-ffd44119896d","Type":"ContainerDied","Data":"c456b1b020d39f8d6730a3ce51250125e1092ab61b4fcd3fe73d0171b62ba963"}
Sep 17 18:04:58.985901 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.985889    2574 scope.go:117] "RemoveContainer" containerID="09c42cc0920ce439091e942d4b1b72b6260670c41d7f0e588e8a0f27809cc40f"
Sep 17 18:04:58.987861 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.987839    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g" event={"ID":"0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67","Type":"ContainerDied","Data":"3ca059c5f0568dcfeb7adbccbb9538602f14f604685485e25144d296b373bd3d"}
Sep 17 18:04:58.987934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.987879    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ca059c5f0568dcfeb7adbccbb9538602f14f604685485e25144d296b373bd3d"
Sep 17 18:04:58.987934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.987881    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lqn7g"
Sep 17 18:04:58.994629 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.994607    2574 scope.go:117] "RemoveContainer" containerID="09c42cc0920ce439091e942d4b1b72b6260670c41d7f0e588e8a0f27809cc40f"
Sep 17 18:04:58.994942 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:04:58.994925    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c42cc0920ce439091e942d4b1b72b6260670c41d7f0e588e8a0f27809cc40f\": container with ID starting with 09c42cc0920ce439091e942d4b1b72b6260670c41d7f0e588e8a0f27809cc40f not found: ID does not exist" containerID="09c42cc0920ce439091e942d4b1b72b6260670c41d7f0e588e8a0f27809cc40f"
Sep 17 18:04:58.995011 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:58.994951    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c42cc0920ce439091e942d4b1b72b6260670c41d7f0e588e8a0f27809cc40f"} err="failed to get container status \"09c42cc0920ce439091e942d4b1b72b6260670c41d7f0e588e8a0f27809cc40f\": rpc error: code = NotFound desc = could not find container \"09c42cc0920ce439091e942d4b1b72b6260670c41d7f0e588e8a0f27809cc40f\": container with ID starting with 09c42cc0920ce439091e942d4b1b72b6260670c41d7f0e588e8a0f27809cc40f not found: ID does not exist"
Sep 17 18:04:59.025668 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:59.025643    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-845cc4bd59-llwmb"]
Sep 17 18:04:59.030387 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:59.030367    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-845cc4bd59-llwmb"]
Sep 17 18:04:59.937266 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:04:59.937233    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e92cb037-a5c6-4139-949d-ffd44119896d" path="/var/lib/kubelet/pods/e92cb037-a5c6-4139-949d-ffd44119896d/volumes"
Sep 17 18:05:03.871482 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:03.871450    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:04.122807 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:04.122725    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:06.154297 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.154253    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mgxb9"]
Sep 17 18:05:06.154688 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.154541    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67" containerName="extract"
Sep 17 18:05:06.154688 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.154552    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67" containerName="extract"
Sep 17 18:05:06.154688 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.154564    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92cb037-a5c6-4139-949d-ffd44119896d" containerName="console"
Sep 17 18:05:06.154688 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.154570    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92cb037-a5c6-4139-949d-ffd44119896d" containerName="console"
Sep 17 18:05:06.154688 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.154577    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67" containerName="util"
Sep 17 18:05:06.154688 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.154583    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67" containerName="util"
Sep 17 18:05:06.154688 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.154589    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67" containerName="pull"
Sep 17 18:05:06.154688 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.154594    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67" containerName="pull"
Sep 17 18:05:06.154688 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.154671    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92cb037-a5c6-4139-949d-ffd44119896d" containerName="console"
Sep 17 18:05:06.154688 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.154680    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a0f4f9b-4db2-45cb-ac8e-a4d4deb30b67" containerName="extract"
Sep 17 18:05:06.157582 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.157566    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mgxb9"
Sep 17 18:05:06.161676 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.161657    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 17 18:05:06.161948 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.161931    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:05:06.162139 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.162123    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-tvk8w\""
Sep 17 18:05:06.175548 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.175509    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mgxb9"]
Sep 17 18:05:06.278230 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.278188    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8f0a773-897f-4763-8d73-560ba9f01108-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-mgxb9\" (UID: \"f8f0a773-897f-4763-8d73-560ba9f01108\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mgxb9"
Sep 17 18:05:06.278410 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.278261    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzx7\" (UniqueName: \"kubernetes.io/projected/f8f0a773-897f-4763-8d73-560ba9f01108-kube-api-access-7gzx7\") pod \"cert-manager-operator-controller-manager-684f79c9b9-mgxb9\" (UID: \"f8f0a773-897f-4763-8d73-560ba9f01108\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mgxb9"
Sep 17 18:05:06.379616 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.379588    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8f0a773-897f-4763-8d73-560ba9f01108-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-mgxb9\" (UID: \"f8f0a773-897f-4763-8d73-560ba9f01108\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mgxb9"
Sep 17 18:05:06.379769 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.379642    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzx7\" (UniqueName: \"kubernetes.io/projected/f8f0a773-897f-4763-8d73-560ba9f01108-kube-api-access-7gzx7\") pod \"cert-manager-operator-controller-manager-684f79c9b9-mgxb9\" (UID: \"f8f0a773-897f-4763-8d73-560ba9f01108\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mgxb9"
Sep 17 18:05:06.379956 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.379937    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8f0a773-897f-4763-8d73-560ba9f01108-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-mgxb9\" (UID: \"f8f0a773-897f-4763-8d73-560ba9f01108\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mgxb9"
Sep 17 18:05:06.388917 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.388894    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzx7\" (UniqueName: \"kubernetes.io/projected/f8f0a773-897f-4763-8d73-560ba9f01108-kube-api-access-7gzx7\") pod \"cert-manager-operator-controller-manager-684f79c9b9-mgxb9\" (UID: \"f8f0a773-897f-4763-8d73-560ba9f01108\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mgxb9"
Sep 17 18:05:06.465904 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.465849    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mgxb9"
Sep 17 18:05:06.739884 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:06.739853    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mgxb9"]
Sep 17 18:05:06.742266 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:05:06.742234    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f0a773_897f_4763_8d73_560ba9f01108.slice/crio-8d62df53fea031b9823c030cad4e56afab2753854fbf2d3de28eed82595a07ef WatchSource:0}: Error finding container 8d62df53fea031b9823c030cad4e56afab2753854fbf2d3de28eed82595a07ef: Status 404 returned error can't find the container with id 8d62df53fea031b9823c030cad4e56afab2753854fbf2d3de28eed82595a07ef
Sep 17 18:05:07.021583 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:07.021547    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mgxb9" event={"ID":"f8f0a773-897f-4763-8d73-560ba9f01108","Type":"ContainerStarted","Data":"8d62df53fea031b9823c030cad4e56afab2753854fbf2d3de28eed82595a07ef"}
Sep 17 18:05:08.107959 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:08.107925    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:08.214765 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:08.214724    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:09.034465 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:09.034416    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mgxb9" event={"ID":"f8f0a773-897f-4763-8d73-560ba9f01108","Type":"ContainerStarted","Data":"5305e62f0d25a1ee16408ca8714f4365c0d5ac166c49a92ab2efc14facaadef5"}
Sep 17 18:05:09.073820 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:09.073764    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-mgxb9" podStartSLOduration=1.396165176 podStartE2EDuration="3.073749399s" totalImagesPullingTime="1.677584223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:09.073132553 +0000 UTC m=+479.373954876" watchObservedRunningTime="2026-09-17 18:05:09.073749399 +0000 UTC m=+479.374571722"
Sep 17 18:05:10.266993 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:10.266963    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:10.447916 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:10.447876    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:12.515232 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:12.515196    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:12.840646 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:12.840560    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:12.936350 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:12.936320    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:13.904219 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:13.904175    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:14.005040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:14.005006    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:14.182503 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:14.182414    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:14.894567 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:14.894535    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"]
Sep 17 18:05:14.897756 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:14.897390    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"
Sep 17 18:05:14.900307 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:14.900286    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lc8mp\""
Sep 17 18:05:14.900501 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:14.900481    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:05:14.901142 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:14.901125    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:05:14.907833 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:14.907815    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"]
Sep 17 18:05:15.047455 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:15.047417    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t\" (UID: \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"
Sep 17 18:05:15.047455 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:15.047461    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t\" (UID: \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"
Sep 17 18:05:15.047681 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:15.047556    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8vtm\" (UniqueName: \"kubernetes.io/projected/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-kube-api-access-t8vtm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t\" (UID: \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"
Sep 17 18:05:15.148923 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:15.148834    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8vtm\" (UniqueName: \"kubernetes.io/projected/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-kube-api-access-t8vtm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t\" (UID: \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"
Sep 17 18:05:15.148923 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:15.148900    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t\" (UID: \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"
Sep 17 18:05:15.148923 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:15.148924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t\" (UID: \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"
Sep 17 18:05:15.149276 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:15.149259    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t\" (UID: \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"
Sep 17 18:05:15.149331 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:15.149312    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t\" (UID: \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"
Sep 17 18:05:15.160029 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:15.160004    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8vtm\" (UniqueName: \"kubernetes.io/projected/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-kube-api-access-t8vtm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t\" (UID: \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"
Sep 17 18:05:15.207848 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:15.207814    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"
Sep 17 18:05:15.484898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:15.484823    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"]
Sep 17 18:05:15.488036 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:05:15.488010    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d13480d_5bd6_4d17_801c_8fc936fb4cc8.slice/crio-fe3dcfee750cc67436262769c1bde614a98915ac5a9d6430d57ad46395913db8 WatchSource:0}: Error finding container fe3dcfee750cc67436262769c1bde614a98915ac5a9d6430d57ad46395913db8: Status 404 returned error can't find the container with id fe3dcfee750cc67436262769c1bde614a98915ac5a9d6430d57ad46395913db8
Sep 17 18:05:16.067861 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:16.067826    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d13480d-5bd6-4d17-801c-8fc936fb4cc8" containerID="be83c1eedb8d4dc57c4c46e246bd33ffc09bdef87c51857b68e7b01c666b15a8" exitCode=0
Sep 17 18:05:16.068294 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:16.067912    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t" event={"ID":"4d13480d-5bd6-4d17-801c-8fc936fb4cc8","Type":"ContainerDied","Data":"be83c1eedb8d4dc57c4c46e246bd33ffc09bdef87c51857b68e7b01c666b15a8"}
Sep 17 18:05:16.068294 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:16.067968    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t" event={"ID":"4d13480d-5bd6-4d17-801c-8fc936fb4cc8","Type":"ContainerStarted","Data":"fe3dcfee750cc67436262769c1bde614a98915ac5a9d6430d57ad46395913db8"}
Sep 17 18:05:17.931028 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:17.930973    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:18.068294 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:18.068264    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:19.083143 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:19.083112    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d13480d-5bd6-4d17-801c-8fc936fb4cc8" containerID="9b985747de5a248f27d3b3b26fb8c1e6fd2004cbcd5481c30540ac503eb92271" exitCode=0
Sep 17 18:05:19.083500 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:19.083156    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t" event={"ID":"4d13480d-5bd6-4d17-801c-8fc936fb4cc8","Type":"ContainerDied","Data":"9b985747de5a248f27d3b3b26fb8c1e6fd2004cbcd5481c30540ac503eb92271"}
Sep 17 18:05:20.090672 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:20.090635    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d13480d-5bd6-4d17-801c-8fc936fb4cc8" containerID="6c108b7cb30114dec0ab5ddd4448ab8c2afbb4ecf77e752ec252ec389a86b58b" exitCode=0
Sep 17 18:05:20.091056 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:20.090671    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t" event={"ID":"4d13480d-5bd6-4d17-801c-8fc936fb4cc8","Type":"ContainerDied","Data":"6c108b7cb30114dec0ab5ddd4448ab8c2afbb4ecf77e752ec252ec389a86b58b"}
Sep 17 18:05:20.119643 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:20.119606    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:21.233425 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:21.233394    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"
Sep 17 18:05:21.299672 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:21.299644    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-util\") pod \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\" (UID: \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\") "
Sep 17 18:05:21.299835 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:21.299694    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-bundle\") pod \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\" (UID: \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\") "
Sep 17 18:05:21.299835 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:21.299748    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8vtm\" (UniqueName: \"kubernetes.io/projected/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-kube-api-access-t8vtm\") pod \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\" (UID: \"4d13480d-5bd6-4d17-801c-8fc936fb4cc8\") "
Sep 17 18:05:21.300071 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:21.300049    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-bundle" (OuterVolumeSpecName: "bundle") pod "4d13480d-5bd6-4d17-801c-8fc936fb4cc8" (UID: "4d13480d-5bd6-4d17-801c-8fc936fb4cc8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:21.302006 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:21.301972    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-kube-api-access-t8vtm" (OuterVolumeSpecName: "kube-api-access-t8vtm") pod "4d13480d-5bd6-4d17-801c-8fc936fb4cc8" (UID: "4d13480d-5bd6-4d17-801c-8fc936fb4cc8"). InnerVolumeSpecName "kube-api-access-t8vtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:21.304428 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:21.304408    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-util" (OuterVolumeSpecName: "util") pod "4d13480d-5bd6-4d17-801c-8fc936fb4cc8" (UID: "4d13480d-5bd6-4d17-801c-8fc936fb4cc8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:21.400884 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:21.400819    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-util\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:05:21.400884 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:21.400842    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-bundle\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:05:21.400884 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:21.400850    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8vtm\" (UniqueName: \"kubernetes.io/projected/4d13480d-5bd6-4d17-801c-8fc936fb4cc8-kube-api-access-t8vtm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:05:22.101331 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:22.101289    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t" event={"ID":"4d13480d-5bd6-4d17-801c-8fc936fb4cc8","Type":"ContainerDied","Data":"fe3dcfee750cc67436262769c1bde614a98915ac5a9d6430d57ad46395913db8"}
Sep 17 18:05:22.101331 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:22.101316    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzb9t"
Sep 17 18:05:22.101534 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:22.101345    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe3dcfee750cc67436262769c1bde614a98915ac5a9d6430d57ad46395913db8"
Sep 17 18:05:24.157137 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:24.157107    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:24.642533 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:24.642480    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:24.742001 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:24.741965    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:26.093704 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.093671    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvpzp"]
Sep 17 18:05:26.094059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.093953    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d13480d-5bd6-4d17-801c-8fc936fb4cc8" containerName="extract"
Sep 17 18:05:26.094059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.093963    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d13480d-5bd6-4d17-801c-8fc936fb4cc8" containerName="extract"
Sep 17 18:05:26.094059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.093978    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d13480d-5bd6-4d17-801c-8fc936fb4cc8" containerName="pull"
Sep 17 18:05:26.094059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.093984    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d13480d-5bd6-4d17-801c-8fc936fb4cc8" containerName="pull"
Sep 17 18:05:26.094059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.093996    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d13480d-5bd6-4d17-801c-8fc936fb4cc8" containerName="util"
Sep 17 18:05:26.094059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.094001    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d13480d-5bd6-4d17-801c-8fc936fb4cc8" containerName="util"
Sep 17 18:05:26.094059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.094053    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d13480d-5bd6-4d17-801c-8fc936fb4cc8" containerName="extract"
Sep 17 18:05:26.097362 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.097341    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvpzp"
Sep 17 18:05:26.100098 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.100078    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:05:26.101303 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.101283    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 18:05:26.101411 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.101283    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-5xrck\""
Sep 17 18:05:26.107304 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.107283    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvpzp"]
Sep 17 18:05:26.131965 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.131941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzbrx\" (UniqueName: \"kubernetes.io/projected/eb0cac6e-e23d-43da-a3c0-e351524c0262-kube-api-access-lzbrx\") pod \"openshift-lws-operator-fd8ccff4c-jvpzp\" (UID: \"eb0cac6e-e23d-43da-a3c0-e351524c0262\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvpzp"
Sep 17 18:05:26.132096 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.131990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb0cac6e-e23d-43da-a3c0-e351524c0262-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jvpzp\" (UID: \"eb0cac6e-e23d-43da-a3c0-e351524c0262\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvpzp"
Sep 17 18:05:26.232840 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.232798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzbrx\" (UniqueName: \"kubernetes.io/projected/eb0cac6e-e23d-43da-a3c0-e351524c0262-kube-api-access-lzbrx\") pod \"openshift-lws-operator-fd8ccff4c-jvpzp\" (UID: \"eb0cac6e-e23d-43da-a3c0-e351524c0262\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvpzp"
Sep 17 18:05:26.233046 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.232871    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb0cac6e-e23d-43da-a3c0-e351524c0262-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jvpzp\" (UID: \"eb0cac6e-e23d-43da-a3c0-e351524c0262\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvpzp"
Sep 17 18:05:26.233197 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.233175    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb0cac6e-e23d-43da-a3c0-e351524c0262-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jvpzp\" (UID: \"eb0cac6e-e23d-43da-a3c0-e351524c0262\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvpzp"
Sep 17 18:05:26.243453 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.243424    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzbrx\" (UniqueName: \"kubernetes.io/projected/eb0cac6e-e23d-43da-a3c0-e351524c0262-kube-api-access-lzbrx\") pod \"openshift-lws-operator-fd8ccff4c-jvpzp\" (UID: \"eb0cac6e-e23d-43da-a3c0-e351524c0262\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvpzp"
Sep 17 18:05:26.414930 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.414851    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvpzp"
Sep 17 18:05:26.697316 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:26.697291    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvpzp"]
Sep 17 18:05:26.699699 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:05:26.699662    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb0cac6e_e23d_43da_a3c0_e351524c0262.slice/crio-8379241148fcd73f3871724cb58efe1df66d75eb026981c09eca896ca30679b4 WatchSource:0}: Error finding container 8379241148fcd73f3871724cb58efe1df66d75eb026981c09eca896ca30679b4: Status 404 returned error can't find the container with id 8379241148fcd73f3871724cb58efe1df66d75eb026981c09eca896ca30679b4
Sep 17 18:05:27.124793 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:27.124759    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvpzp" event={"ID":"eb0cac6e-e23d-43da-a3c0-e351524c0262","Type":"ContainerStarted","Data":"8379241148fcd73f3871724cb58efe1df66d75eb026981c09eca896ca30679b4"}
Sep 17 18:05:29.135496 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:29.135459    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvpzp" event={"ID":"eb0cac6e-e23d-43da-a3c0-e351524c0262","Type":"ContainerStarted","Data":"cf6fe6d1653d59164401d345924587dfa8be1805c0c150b4636965ba05bd8569"}
Sep 17 18:05:29.157179 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:29.157141    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jvpzp" podStartSLOduration=1.241879671 podStartE2EDuration="3.157127564s" totalImagesPullingTime="1.915247893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:29.155196629 +0000 UTC m=+499.456018953" watchObservedRunningTime="2026-09-17 18:05:29.157127564 +0000 UTC m=+499.457949888"
Sep 17 18:05:32.204003 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:32.203969    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:37.355937 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.355901    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"]
Sep 17 18:05:37.356343 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.356323    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:37.358670 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.358655    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.361972 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.361954    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 18:05:37.362108 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.362023    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 18:05:37.362886 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.362861    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:05:37.362886 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.362871    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 18:05:37.363071 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.362901    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 18:05:37.363071 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.362870    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h9954\""
Sep 17 18:05:37.363806 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.363786    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 18:05:37.371710 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.371689    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"]
Sep 17 18:05:37.411756 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.411735    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.411847 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.411765    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhxlz\" (UniqueName: \"kubernetes.io/projected/704e2be6-21b0-420a-afb0-06bd98d985ed-kube-api-access-lhxlz\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.411847 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.411794    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.411847 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.411831    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.411952 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.411892    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/704e2be6-21b0-420a-afb0-06bd98d985ed-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.411952 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.411922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.411952 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.411938    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.437272 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.437245    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:37.512624 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.512599    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/704e2be6-21b0-420a-afb0-06bd98d985ed-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.512720 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.512640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.512720 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.512664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.512720 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.512704    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.512867 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.512731    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhxlz\" (UniqueName: \"kubernetes.io/projected/704e2be6-21b0-420a-afb0-06bd98d985ed-kube-api-access-lhxlz\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.512867 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.512763    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.512867 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.512810    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.513345 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.513317    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.514930 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.514911    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/704e2be6-21b0-420a-afb0-06bd98d985ed-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.515235 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.515218    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.515362 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.515347    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.515398 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.515352    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.521598 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.521581    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.525640 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.525616    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhxlz\" (UniqueName: \"kubernetes.io/projected/704e2be6-21b0-420a-afb0-06bd98d985ed-kube-api-access-lhxlz\") pod \"istiod-openshift-gateway-7b7995d5cd-bzf22\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.667884 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.667834    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:37.942028 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:37.942006    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"]
Sep 17 18:05:37.943879 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:05:37.943852    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704e2be6_21b0_420a_afb0_06bd98d985ed.slice/crio-8f27d8b2ae12e780ca164b73dfa4cbd40c04c8577f251fb4a8b6618fad9153cb WatchSource:0}: Error finding container 8f27d8b2ae12e780ca164b73dfa4cbd40c04c8577f251fb4a8b6618fad9153cb: Status 404 returned error can't find the container with id 8f27d8b2ae12e780ca164b73dfa4cbd40c04c8577f251fb4a8b6618fad9153cb
Sep 17 18:05:38.177027 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:38.176996    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22" event={"ID":"704e2be6-21b0-420a-afb0-06bd98d985ed","Type":"ContainerStarted","Data":"8f27d8b2ae12e780ca164b73dfa4cbd40c04c8577f251fb4a8b6618fad9153cb"}
Sep 17 18:05:38.538215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:38.538171    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:38.627987 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:38.627958    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:38.775177 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:38.775149    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:40.463469 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:40.463419    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 18:05:40.463852 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:40.463509    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 18:05:40.486744 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:40.486723    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:40.587102 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:40.587067    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:41.041640 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:41.041612    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:41.140491 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:41.140459    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:41.195272 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:41.195233    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22" event={"ID":"704e2be6-21b0-420a-afb0-06bd98d985ed","Type":"ContainerStarted","Data":"df3aed60fa51feebfc0b1a6c8061ebcdeda3822ae8942f3a829e212db2ec2701"}
Sep 17 18:05:41.195437 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:41.195397    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:41.197304 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:41.197276    2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-bzf22 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 18:05:41.197424 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:41.197333    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22" podUID="704e2be6-21b0-420a-afb0-06bd98d985ed" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:05:41.222988 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:41.222933    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22" podStartSLOduration=1.70549136 podStartE2EDuration="4.222914582s" totalImagesPullingTime="2.517423222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:41.220307731 +0000 UTC m=+511.521130070" watchObservedRunningTime="2026-09-17 18:05:41.222914582 +0000 UTC m=+511.523736905"
Sep 17 18:05:42.201047 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:42.201021    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:05:43.763014 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:43.762979    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:43.855374 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:43.855340    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:46.914319 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:46.914286    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:49.316139 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:49.316105    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:49.446908 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:49.446878    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:52.449475 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:52.449398    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:52.551431 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:52.551398    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:54.607742 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.607708    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"]
Sep 17 18:05:54.610942 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.610925    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"
Sep 17 18:05:54.613820 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.613799    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:05:54.614527 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.614502    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lc8mp\""
Sep 17 18:05:54.614573 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.614526    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:05:54.622254 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.622233    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"]
Sep 17 18:05:54.704544 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.704483    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"]
Sep 17 18:05:54.707779 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.707758    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"
Sep 17 18:05:54.719755 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.719734    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"]
Sep 17 18:05:54.758100 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.758070    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0723283-880f-49a5-8324-34094c7af3a0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s\" (UID: \"f0723283-880f-49a5-8324-34094c7af3a0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"
Sep 17 18:05:54.758268 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.758132    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a93d30ef-7787-42c6-8d0e-8543dd1971fb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d\" (UID: \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"
Sep 17 18:05:54.758268 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.758186    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvx77\" (UniqueName: \"kubernetes.io/projected/a93d30ef-7787-42c6-8d0e-8543dd1971fb-kube-api-access-jvx77\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d\" (UID: \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"
Sep 17 18:05:54.758268 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.758247    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6plr4\" (UniqueName: \"kubernetes.io/projected/f0723283-880f-49a5-8324-34094c7af3a0-kube-api-access-6plr4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s\" (UID: \"f0723283-880f-49a5-8324-34094c7af3a0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"
Sep 17 18:05:54.758398 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.758276    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a93d30ef-7787-42c6-8d0e-8543dd1971fb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d\" (UID: \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"
Sep 17 18:05:54.758398 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.758300    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0723283-880f-49a5-8324-34094c7af3a0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s\" (UID: \"f0723283-880f-49a5-8324-34094c7af3a0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"
Sep 17 18:05:54.817531 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.817492    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"]
Sep 17 18:05:54.820710 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.820695    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"
Sep 17 18:05:54.833725 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.833698    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"]
Sep 17 18:05:54.858914 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.858836    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0723283-880f-49a5-8324-34094c7af3a0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s\" (UID: \"f0723283-880f-49a5-8324-34094c7af3a0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"
Sep 17 18:05:54.859061 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.858919    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a93d30ef-7787-42c6-8d0e-8543dd1971fb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d\" (UID: \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"
Sep 17 18:05:54.859061 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.858952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvx77\" (UniqueName: \"kubernetes.io/projected/a93d30ef-7787-42c6-8d0e-8543dd1971fb-kube-api-access-jvx77\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d\" (UID: \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"
Sep 17 18:05:54.859061 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.858998    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6plr4\" (UniqueName: \"kubernetes.io/projected/f0723283-880f-49a5-8324-34094c7af3a0-kube-api-access-6plr4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s\" (UID: \"f0723283-880f-49a5-8324-34094c7af3a0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"
Sep 17 18:05:54.859061 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.859032    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a93d30ef-7787-42c6-8d0e-8543dd1971fb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d\" (UID: \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"
Sep 17 18:05:54.859061 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.859060    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0723283-880f-49a5-8324-34094c7af3a0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s\" (UID: \"f0723283-880f-49a5-8324-34094c7af3a0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"
Sep 17 18:05:54.859319 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.859243    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0723283-880f-49a5-8324-34094c7af3a0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s\" (UID: \"f0723283-880f-49a5-8324-34094c7af3a0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"
Sep 17 18:05:54.859319 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.859274    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a93d30ef-7787-42c6-8d0e-8543dd1971fb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d\" (UID: \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"
Sep 17 18:05:54.859411 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.859354    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0723283-880f-49a5-8324-34094c7af3a0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s\" (UID: \"f0723283-880f-49a5-8324-34094c7af3a0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"
Sep 17 18:05:54.859547 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.859500    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a93d30ef-7787-42c6-8d0e-8543dd1971fb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d\" (UID: \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"
Sep 17 18:05:54.871886 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.871856    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6plr4\" (UniqueName: \"kubernetes.io/projected/f0723283-880f-49a5-8324-34094c7af3a0-kube-api-access-6plr4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s\" (UID: \"f0723283-880f-49a5-8324-34094c7af3a0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"
Sep 17 18:05:54.873028 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.873011    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvx77\" (UniqueName: \"kubernetes.io/projected/a93d30ef-7787-42c6-8d0e-8543dd1971fb-kube-api-access-jvx77\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d\" (UID: \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"
Sep 17 18:05:54.919565 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.919537    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"
Sep 17 18:05:54.960330 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.960296    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c47a502c-2108-4f57-a7f2-90d7af11b5bd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq\" (UID: \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"
Sep 17 18:05:54.960481 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.960345    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5kvb\" (UniqueName: \"kubernetes.io/projected/c47a502c-2108-4f57-a7f2-90d7af11b5bd-kube-api-access-s5kvb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq\" (UID: \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"
Sep 17 18:05:54.960481 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.960432    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c47a502c-2108-4f57-a7f2-90d7af11b5bd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq\" (UID: \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"
Sep 17 18:05:54.972607 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.972576    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"]
Sep 17 18:05:54.978776 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.978757    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"
Sep 17 18:05:54.998162 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:54.998132    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"]
Sep 17 18:05:55.016557 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.016504    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"
Sep 17 18:05:55.061613 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.061571    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c47a502c-2108-4f57-a7f2-90d7af11b5bd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq\" (UID: \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"
Sep 17 18:05:55.062082 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.062041    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c47a502c-2108-4f57-a7f2-90d7af11b5bd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq\" (UID: \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"
Sep 17 18:05:55.062696 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.062669    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kvb\" (UniqueName: \"kubernetes.io/projected/c47a502c-2108-4f57-a7f2-90d7af11b5bd-kube-api-access-s5kvb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq\" (UID: \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"
Sep 17 18:05:55.062792 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.062773    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75d15832-833f-49e8-9cab-914b54b3b914-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk\" (UID: \"75d15832-833f-49e8-9cab-914b54b3b914\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"
Sep 17 18:05:55.063412 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.062863    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c47a502c-2108-4f57-a7f2-90d7af11b5bd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq\" (UID: \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"
Sep 17 18:05:55.063412 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.062920    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75d15832-833f-49e8-9cab-914b54b3b914-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk\" (UID: \"75d15832-833f-49e8-9cab-914b54b3b914\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"
Sep 17 18:05:55.063412 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.063118    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnwvb\" (UniqueName: \"kubernetes.io/projected/75d15832-833f-49e8-9cab-914b54b3b914-kube-api-access-fnwvb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk\" (UID: \"75d15832-833f-49e8-9cab-914b54b3b914\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"
Sep 17 18:05:55.063412 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.063352    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c47a502c-2108-4f57-a7f2-90d7af11b5bd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq\" (UID: \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"
Sep 17 18:05:55.077254 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.077206    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5kvb\" (UniqueName: \"kubernetes.io/projected/c47a502c-2108-4f57-a7f2-90d7af11b5bd-kube-api-access-s5kvb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq\" (UID: \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"
Sep 17 18:05:55.129372 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.129289    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"
Sep 17 18:05:55.163899 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.163860    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75d15832-833f-49e8-9cab-914b54b3b914-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk\" (UID: \"75d15832-833f-49e8-9cab-914b54b3b914\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"
Sep 17 18:05:55.164057 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.163909    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75d15832-833f-49e8-9cab-914b54b3b914-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk\" (UID: \"75d15832-833f-49e8-9cab-914b54b3b914\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"
Sep 17 18:05:55.164057 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.163938    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnwvb\" (UniqueName: \"kubernetes.io/projected/75d15832-833f-49e8-9cab-914b54b3b914-kube-api-access-fnwvb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk\" (UID: \"75d15832-833f-49e8-9cab-914b54b3b914\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"
Sep 17 18:05:55.164378 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.164360    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75d15832-833f-49e8-9cab-914b54b3b914-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk\" (UID: \"75d15832-833f-49e8-9cab-914b54b3b914\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"
Sep 17 18:05:55.164436 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.164413    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75d15832-833f-49e8-9cab-914b54b3b914-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk\" (UID: \"75d15832-833f-49e8-9cab-914b54b3b914\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"
Sep 17 18:05:55.174308 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.174284    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnwvb\" (UniqueName: \"kubernetes.io/projected/75d15832-833f-49e8-9cab-914b54b3b914-kube-api-access-fnwvb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk\" (UID: \"75d15832-833f-49e8-9cab-914b54b3b914\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"
Sep 17 18:05:55.197860 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.197830    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"]
Sep 17 18:05:55.199023 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:05:55.198997    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93d30ef_7787_42c6_8d0e_8543dd1971fb.slice/crio-e24d6cc951c9282dee06a8ea4b15d841c488cea89343e1d83af628778bd37234 WatchSource:0}: Error finding container e24d6cc951c9282dee06a8ea4b15d841c488cea89343e1d83af628778bd37234: Status 404 returned error can't find the container with id e24d6cc951c9282dee06a8ea4b15d841c488cea89343e1d83af628778bd37234
Sep 17 18:05:55.259500 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.259473    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d" event={"ID":"a93d30ef-7787-42c6-8d0e-8543dd1971fb","Type":"ContainerStarted","Data":"e24d6cc951c9282dee06a8ea4b15d841c488cea89343e1d83af628778bd37234"}
Sep 17 18:05:55.290326 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.290281    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"
Sep 17 18:05:55.298559 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.296572    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"]
Sep 17 18:05:55.303329 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:05:55.303301    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0723283_880f_49a5_8324_34094c7af3a0.slice/crio-dea52008129d887cbb1b3959488cdc945819bfc664bd93b587da26353adeba34 WatchSource:0}: Error finding container dea52008129d887cbb1b3959488cdc945819bfc664bd93b587da26353adeba34: Status 404 returned error can't find the container with id dea52008129d887cbb1b3959488cdc945819bfc664bd93b587da26353adeba34
Sep 17 18:05:55.402441 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.402409    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"]
Sep 17 18:05:55.405119 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:05:55.405094    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47a502c_2108_4f57_a7f2_90d7af11b5bd.slice/crio-c23dd0391caf0ab22cdfdf272a588323d6379a67bffc62064e55a2f75554ac5e WatchSource:0}: Error finding container c23dd0391caf0ab22cdfdf272a588323d6379a67bffc62064e55a2f75554ac5e: Status 404 returned error can't find the container with id c23dd0391caf0ab22cdfdf272a588323d6379a67bffc62064e55a2f75554ac5e
Sep 17 18:05:55.572466 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:55.572445    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"]
Sep 17 18:05:55.630306 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:05:55.630250    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d15832_833f_49e8_9cab_914b54b3b914.slice/crio-7043c15aebde2f679dfa960ee7761f99227a5019d5057949a87062d3ec479be3 WatchSource:0}: Error finding container 7043c15aebde2f679dfa960ee7761f99227a5019d5057949a87062d3ec479be3: Status 404 returned error can't find the container with id 7043c15aebde2f679dfa960ee7761f99227a5019d5057949a87062d3ec479be3
Sep 17 18:05:56.265562 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:56.265507    2574 generic.go:356] "Generic (PLEG): container finished" podID="f0723283-880f-49a5-8324-34094c7af3a0" containerID="bfa053826901a44151f95b46d7d02d8640b18b77a8b945dc87bf681b3560e443" exitCode=0
Sep 17 18:05:56.265751 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:56.265612    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s" event={"ID":"f0723283-880f-49a5-8324-34094c7af3a0","Type":"ContainerDied","Data":"bfa053826901a44151f95b46d7d02d8640b18b77a8b945dc87bf681b3560e443"}
Sep 17 18:05:56.265751 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:56.265672    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s" event={"ID":"f0723283-880f-49a5-8324-34094c7af3a0","Type":"ContainerStarted","Data":"dea52008129d887cbb1b3959488cdc945819bfc664bd93b587da26353adeba34"}
Sep 17 18:05:56.267211 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:56.267193    2574 generic.go:356] "Generic (PLEG): container finished" podID="a93d30ef-7787-42c6-8d0e-8543dd1971fb" containerID="a3b940931cf3c835d19142856907f1c5b6de21f5e135e6b1ff2920dc1c924626" exitCode=0
Sep 17 18:05:56.267273 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:56.267255    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d" event={"ID":"a93d30ef-7787-42c6-8d0e-8543dd1971fb","Type":"ContainerDied","Data":"a3b940931cf3c835d19142856907f1c5b6de21f5e135e6b1ff2920dc1c924626"}
Sep 17 18:05:56.268929 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:56.268909    2574 generic.go:356] "Generic (PLEG): container finished" podID="75d15832-833f-49e8-9cab-914b54b3b914" containerID="459ae2577686401aa1dbb94d9f638b5db433ab70fcecc9236285ca638b98b610" exitCode=0
Sep 17 18:05:56.269055 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:56.269032    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk" event={"ID":"75d15832-833f-49e8-9cab-914b54b3b914","Type":"ContainerDied","Data":"459ae2577686401aa1dbb94d9f638b5db433ab70fcecc9236285ca638b98b610"}
Sep 17 18:05:56.269162 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:56.269079    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk" event={"ID":"75d15832-833f-49e8-9cab-914b54b3b914","Type":"ContainerStarted","Data":"7043c15aebde2f679dfa960ee7761f99227a5019d5057949a87062d3ec479be3"}
Sep 17 18:05:56.270742 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:56.270721    2574 generic.go:356] "Generic (PLEG): container finished" podID="c47a502c-2108-4f57-a7f2-90d7af11b5bd" containerID="a11ede35a627ec75effd0846bdb77ba43584980aae38daa9b5a5dc5184917871" exitCode=0
Sep 17 18:05:56.270816 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:56.270796    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq" event={"ID":"c47a502c-2108-4f57-a7f2-90d7af11b5bd","Type":"ContainerDied","Data":"a11ede35a627ec75effd0846bdb77ba43584980aae38daa9b5a5dc5184917871"}
Sep 17 18:05:56.270862 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:56.270830    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq" event={"ID":"c47a502c-2108-4f57-a7f2-90d7af11b5bd","Type":"ContainerStarted","Data":"c23dd0391caf0ab22cdfdf272a588323d6379a67bffc62064e55a2f75554ac5e"}
Sep 17 18:05:57.281269 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:57.278540    2574 generic.go:356] "Generic (PLEG): container finished" podID="75d15832-833f-49e8-9cab-914b54b3b914" containerID="286a894ad8c454c54443135b8015d3cd3ba27f3830f84bce99bc9549890e2939" exitCode=0
Sep 17 18:05:57.281269 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:57.278589    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk" event={"ID":"75d15832-833f-49e8-9cab-914b54b3b914","Type":"ContainerDied","Data":"286a894ad8c454c54443135b8015d3cd3ba27f3830f84bce99bc9549890e2939"}
Sep 17 18:05:58.285697 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:58.285663    2574 generic.go:356] "Generic (PLEG): container finished" podID="f0723283-880f-49a5-8324-34094c7af3a0" containerID="482dd3c08e5e69997d188a7c5806d24bc880f1ada4a1c9ac525e171fbca928b2" exitCode=0
Sep 17 18:05:58.286116 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:58.285750    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s" event={"ID":"f0723283-880f-49a5-8324-34094c7af3a0","Type":"ContainerDied","Data":"482dd3c08e5e69997d188a7c5806d24bc880f1ada4a1c9ac525e171fbca928b2"}
Sep 17 18:05:58.287756 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:58.287735    2574 generic.go:356] "Generic (PLEG): container finished" podID="a93d30ef-7787-42c6-8d0e-8543dd1971fb" containerID="a01ded1731d43af3acd7c1e995a54c1c0147050e6845246c398b0ec4bb04362b" exitCode=0
Sep 17 18:05:58.287876 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:58.287811    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d" event={"ID":"a93d30ef-7787-42c6-8d0e-8543dd1971fb","Type":"ContainerDied","Data":"a01ded1731d43af3acd7c1e995a54c1c0147050e6845246c398b0ec4bb04362b"}
Sep 17 18:05:58.290116 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:58.290093    2574 generic.go:356] "Generic (PLEG): container finished" podID="75d15832-833f-49e8-9cab-914b54b3b914" containerID="d07aeef301928f76d05cc569ee1dab25f7c62c4aaecbd918fd552e086ee12b1d" exitCode=0
Sep 17 18:05:58.290208 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:58.290171    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk" event={"ID":"75d15832-833f-49e8-9cab-914b54b3b914","Type":"ContainerDied","Data":"d07aeef301928f76d05cc569ee1dab25f7c62c4aaecbd918fd552e086ee12b1d"}
Sep 17 18:05:59.297140 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:59.297107    2574 generic.go:356] "Generic (PLEG): container finished" podID="a93d30ef-7787-42c6-8d0e-8543dd1971fb" containerID="40a2872753d9f7c123c0477881c0d22a75bd2abbd9e75798dd051d621a8aeec5" exitCode=0
Sep 17 18:05:59.297599 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:59.297198    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d" event={"ID":"a93d30ef-7787-42c6-8d0e-8543dd1971fb","Type":"ContainerDied","Data":"40a2872753d9f7c123c0477881c0d22a75bd2abbd9e75798dd051d621a8aeec5"}
Sep 17 18:05:59.299391 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:59.299367    2574 generic.go:356] "Generic (PLEG): container finished" podID="f0723283-880f-49a5-8324-34094c7af3a0" containerID="e3b2404c9b053d0f5a7e5a867684278b916c4011d61161d53b91871c78a689e7" exitCode=0
Sep 17 18:05:59.299498 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:59.299441    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s" event={"ID":"f0723283-880f-49a5-8324-34094c7af3a0","Type":"ContainerDied","Data":"e3b2404c9b053d0f5a7e5a867684278b916c4011d61161d53b91871c78a689e7"}
Sep 17 18:05:59.427778 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:59.427757    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"
Sep 17 18:05:59.502412 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:59.502383    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75d15832-833f-49e8-9cab-914b54b3b914-bundle\") pod \"75d15832-833f-49e8-9cab-914b54b3b914\" (UID: \"75d15832-833f-49e8-9cab-914b54b3b914\") "
Sep 17 18:05:59.502629 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:59.502442    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75d15832-833f-49e8-9cab-914b54b3b914-util\") pod \"75d15832-833f-49e8-9cab-914b54b3b914\" (UID: \"75d15832-833f-49e8-9cab-914b54b3b914\") "
Sep 17 18:05:59.502629 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:59.502475    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnwvb\" (UniqueName: \"kubernetes.io/projected/75d15832-833f-49e8-9cab-914b54b3b914-kube-api-access-fnwvb\") pod \"75d15832-833f-49e8-9cab-914b54b3b914\" (UID: \"75d15832-833f-49e8-9cab-914b54b3b914\") "
Sep 17 18:05:59.503215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:59.503175    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d15832-833f-49e8-9cab-914b54b3b914-bundle" (OuterVolumeSpecName: "bundle") pod "75d15832-833f-49e8-9cab-914b54b3b914" (UID: "75d15832-833f-49e8-9cab-914b54b3b914"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:59.504700 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:59.504676    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d15832-833f-49e8-9cab-914b54b3b914-kube-api-access-fnwvb" (OuterVolumeSpecName: "kube-api-access-fnwvb") pod "75d15832-833f-49e8-9cab-914b54b3b914" (UID: "75d15832-833f-49e8-9cab-914b54b3b914"). InnerVolumeSpecName "kube-api-access-fnwvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:59.508132 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:59.508099    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d15832-833f-49e8-9cab-914b54b3b914-util" (OuterVolumeSpecName: "util") pod "75d15832-833f-49e8-9cab-914b54b3b914" (UID: "75d15832-833f-49e8-9cab-914b54b3b914"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:59.603369 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:59.603293    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75d15832-833f-49e8-9cab-914b54b3b914-bundle\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:05:59.603369 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:59.603328    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75d15832-833f-49e8-9cab-914b54b3b914-util\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:05:59.603369 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:05:59.603337    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnwvb\" (UniqueName: \"kubernetes.io/projected/75d15832-833f-49e8-9cab-914b54b3b914-kube-api-access-fnwvb\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:06:00.307353 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.307324    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk"
Sep 17 18:06:00.307738 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.307354    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqjxjk" event={"ID":"75d15832-833f-49e8-9cab-914b54b3b914","Type":"ContainerDied","Data":"7043c15aebde2f679dfa960ee7761f99227a5019d5057949a87062d3ec479be3"}
Sep 17 18:06:00.307738 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.307403    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7043c15aebde2f679dfa960ee7761f99227a5019d5057949a87062d3ec479be3"
Sep 17 18:06:00.444494 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.444466    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"
Sep 17 18:06:00.479318 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.479293    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"
Sep 17 18:06:00.510435 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.510404    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvx77\" (UniqueName: \"kubernetes.io/projected/a93d30ef-7787-42c6-8d0e-8543dd1971fb-kube-api-access-jvx77\") pod \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\" (UID: \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\") "
Sep 17 18:06:00.510665 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.510484    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a93d30ef-7787-42c6-8d0e-8543dd1971fb-bundle\") pod \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\" (UID: \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\") "
Sep 17 18:06:00.510665 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.510526    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a93d30ef-7787-42c6-8d0e-8543dd1971fb-util\") pod \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\" (UID: \"a93d30ef-7787-42c6-8d0e-8543dd1971fb\") "
Sep 17 18:06:00.511715 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.511634    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93d30ef-7787-42c6-8d0e-8543dd1971fb-bundle" (OuterVolumeSpecName: "bundle") pod "a93d30ef-7787-42c6-8d0e-8543dd1971fb" (UID: "a93d30ef-7787-42c6-8d0e-8543dd1971fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:00.513881 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.513852    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93d30ef-7787-42c6-8d0e-8543dd1971fb-kube-api-access-jvx77" (OuterVolumeSpecName: "kube-api-access-jvx77") pod "a93d30ef-7787-42c6-8d0e-8543dd1971fb" (UID: "a93d30ef-7787-42c6-8d0e-8543dd1971fb"). InnerVolumeSpecName "kube-api-access-jvx77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:06:00.611628 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.611556    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6plr4\" (UniqueName: \"kubernetes.io/projected/f0723283-880f-49a5-8324-34094c7af3a0-kube-api-access-6plr4\") pod \"f0723283-880f-49a5-8324-34094c7af3a0\" (UID: \"f0723283-880f-49a5-8324-34094c7af3a0\") "
Sep 17 18:06:00.611758 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.611663    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0723283-880f-49a5-8324-34094c7af3a0-bundle\") pod \"f0723283-880f-49a5-8324-34094c7af3a0\" (UID: \"f0723283-880f-49a5-8324-34094c7af3a0\") "
Sep 17 18:06:00.611758 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.611684    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0723283-880f-49a5-8324-34094c7af3a0-util\") pod \"f0723283-880f-49a5-8324-34094c7af3a0\" (UID: \"f0723283-880f-49a5-8324-34094c7af3a0\") "
Sep 17 18:06:00.611855 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.611809    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvx77\" (UniqueName: \"kubernetes.io/projected/a93d30ef-7787-42c6-8d0e-8543dd1971fb-kube-api-access-jvx77\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:06:00.611855 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.611820    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a93d30ef-7787-42c6-8d0e-8543dd1971fb-bundle\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:06:00.612167 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.612129    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0723283-880f-49a5-8324-34094c7af3a0-bundle" (OuterVolumeSpecName: "bundle") pod "f0723283-880f-49a5-8324-34094c7af3a0" (UID: "f0723283-880f-49a5-8324-34094c7af3a0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:00.613821 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.613797    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0723283-880f-49a5-8324-34094c7af3a0-kube-api-access-6plr4" (OuterVolumeSpecName: "kube-api-access-6plr4") pod "f0723283-880f-49a5-8324-34094c7af3a0" (UID: "f0723283-880f-49a5-8324-34094c7af3a0"). InnerVolumeSpecName "kube-api-access-6plr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:06:00.712704 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.712675    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0723283-880f-49a5-8324-34094c7af3a0-bundle\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:06:00.712704 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.712700    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6plr4\" (UniqueName: \"kubernetes.io/projected/f0723283-880f-49a5-8324-34094c7af3a0-kube-api-access-6plr4\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:06:00.819475 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.819436    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0723283-880f-49a5-8324-34094c7af3a0-util" (OuterVolumeSpecName: "util") pod "f0723283-880f-49a5-8324-34094c7af3a0" (UID: "f0723283-880f-49a5-8324-34094c7af3a0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:00.827131 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.827105    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93d30ef-7787-42c6-8d0e-8543dd1971fb-util" (OuterVolumeSpecName: "util") pod "a93d30ef-7787-42c6-8d0e-8543dd1971fb" (UID: "a93d30ef-7787-42c6-8d0e-8543dd1971fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:00.914848 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.914769    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a93d30ef-7787-42c6-8d0e-8543dd1971fb-util\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:06:00.914848 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:00.914803    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0723283-880f-49a5-8324-34094c7af3a0-util\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:06:01.313642 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:01.313617    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s"
Sep 17 18:06:01.314031 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:01.313609    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lmj5s" event={"ID":"f0723283-880f-49a5-8324-34094c7af3a0","Type":"ContainerDied","Data":"dea52008129d887cbb1b3959488cdc945819bfc664bd93b587da26353adeba34"}
Sep 17 18:06:01.314031 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:01.313757    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dea52008129d887cbb1b3959488cdc945819bfc664bd93b587da26353adeba34"
Sep 17 18:06:01.315445 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:01.315425    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d" event={"ID":"a93d30ef-7787-42c6-8d0e-8543dd1971fb","Type":"ContainerDied","Data":"e24d6cc951c9282dee06a8ea4b15d841c488cea89343e1d83af628778bd37234"}
Sep 17 18:06:01.315445 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:01.315456    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e24d6cc951c9282dee06a8ea4b15d841c488cea89343e1d83af628778bd37234"
Sep 17 18:06:01.315605 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:01.315458    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnmq5d"
Sep 17 18:06:02.321801 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:02.321769    2574 generic.go:356] "Generic (PLEG): container finished" podID="c47a502c-2108-4f57-a7f2-90d7af11b5bd" containerID="c865d5b5029a832006f12201d0253216ddfdc11b45bfc16375536cb2bbf9b8ef" exitCode=0
Sep 17 18:06:02.322168 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:02.321810    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq" event={"ID":"c47a502c-2108-4f57-a7f2-90d7af11b5bd","Type":"ContainerDied","Data":"c865d5b5029a832006f12201d0253216ddfdc11b45bfc16375536cb2bbf9b8ef"}
Sep 17 18:06:03.328713 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:03.328684    2574 generic.go:356] "Generic (PLEG): container finished" podID="c47a502c-2108-4f57-a7f2-90d7af11b5bd" containerID="e2557284011a7a762b9245c6f12c2e2337b81e9f6c4de4a815e1f6f07774b119" exitCode=0
Sep 17 18:06:03.329159 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:03.328753    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq" event={"ID":"c47a502c-2108-4f57-a7f2-90d7af11b5bd","Type":"ContainerDied","Data":"e2557284011a7a762b9245c6f12c2e2337b81e9f6c4de4a815e1f6f07774b119"}
Sep 17 18:06:04.462185 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:04.462161    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"
Sep 17 18:06:04.544191 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:04.544149    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c47a502c-2108-4f57-a7f2-90d7af11b5bd-bundle\") pod \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\" (UID: \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\") "
Sep 17 18:06:04.544191 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:04.544201    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c47a502c-2108-4f57-a7f2-90d7af11b5bd-util\") pod \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\" (UID: \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\") "
Sep 17 18:06:04.544416 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:04.544237    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5kvb\" (UniqueName: \"kubernetes.io/projected/c47a502c-2108-4f57-a7f2-90d7af11b5bd-kube-api-access-s5kvb\") pod \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\" (UID: \"c47a502c-2108-4f57-a7f2-90d7af11b5bd\") "
Sep 17 18:06:04.544634 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:04.544610    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47a502c-2108-4f57-a7f2-90d7af11b5bd-bundle" (OuterVolumeSpecName: "bundle") pod "c47a502c-2108-4f57-a7f2-90d7af11b5bd" (UID: "c47a502c-2108-4f57-a7f2-90d7af11b5bd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:04.546486 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:04.546459    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47a502c-2108-4f57-a7f2-90d7af11b5bd-kube-api-access-s5kvb" (OuterVolumeSpecName: "kube-api-access-s5kvb") pod "c47a502c-2108-4f57-a7f2-90d7af11b5bd" (UID: "c47a502c-2108-4f57-a7f2-90d7af11b5bd"). InnerVolumeSpecName "kube-api-access-s5kvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:06:04.548701 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:04.548678    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47a502c-2108-4f57-a7f2-90d7af11b5bd-util" (OuterVolumeSpecName: "util") pod "c47a502c-2108-4f57-a7f2-90d7af11b5bd" (UID: "c47a502c-2108-4f57-a7f2-90d7af11b5bd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:04.644818 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:04.644757    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c47a502c-2108-4f57-a7f2-90d7af11b5bd-bundle\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:06:04.644818 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:04.644779    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c47a502c-2108-4f57-a7f2-90d7af11b5bd-util\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:06:04.644818 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:04.644789    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5kvb\" (UniqueName: \"kubernetes.io/projected/c47a502c-2108-4f57-a7f2-90d7af11b5bd-kube-api-access-s5kvb\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:06:05.339680 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:05.339647    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq"
Sep 17 18:06:05.339680 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:05.339670    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6bhq" event={"ID":"c47a502c-2108-4f57-a7f2-90d7af11b5bd","Type":"ContainerDied","Data":"c23dd0391caf0ab22cdfdf272a588323d6379a67bffc62064e55a2f75554ac5e"}
Sep 17 18:06:05.339877 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:05.339733    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c23dd0391caf0ab22cdfdf272a588323d6379a67bffc62064e55a2f75554ac5e"
Sep 17 18:06:08.200440 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:08.200395    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:08.277486 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:08.277438    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:14.506208 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:14.506167    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:15.270671 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:15.270636    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:17.798391 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798355    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-5sq9k"]
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798692    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d15832-833f-49e8-9cab-914b54b3b914" containerName="extract"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798704    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d15832-833f-49e8-9cab-914b54b3b914" containerName="extract"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798717    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0723283-880f-49a5-8324-34094c7af3a0" containerName="util"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798722    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0723283-880f-49a5-8324-34094c7af3a0" containerName="util"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798731    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d15832-833f-49e8-9cab-914b54b3b914" containerName="pull"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798736    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d15832-833f-49e8-9cab-914b54b3b914" containerName="pull"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798744    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47a502c-2108-4f57-a7f2-90d7af11b5bd" containerName="pull"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798748    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47a502c-2108-4f57-a7f2-90d7af11b5bd" containerName="pull"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798757    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47a502c-2108-4f57-a7f2-90d7af11b5bd" containerName="util"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798761    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47a502c-2108-4f57-a7f2-90d7af11b5bd" containerName="util"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798767    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93d30ef-7787-42c6-8d0e-8543dd1971fb" containerName="pull"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798771    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93d30ef-7787-42c6-8d0e-8543dd1971fb" containerName="pull"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798776    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0723283-880f-49a5-8324-34094c7af3a0" containerName="extract"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798780    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0723283-880f-49a5-8324-34094c7af3a0" containerName="extract"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798786    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93d30ef-7787-42c6-8d0e-8543dd1971fb" containerName="extract"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798790    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93d30ef-7787-42c6-8d0e-8543dd1971fb" containerName="extract"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798795    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47a502c-2108-4f57-a7f2-90d7af11b5bd" containerName="extract"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798800    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47a502c-2108-4f57-a7f2-90d7af11b5bd" containerName="extract"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798807    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d15832-833f-49e8-9cab-914b54b3b914" containerName="util"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798812    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d15832-833f-49e8-9cab-914b54b3b914" containerName="util"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798819    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93d30ef-7787-42c6-8d0e-8543dd1971fb" containerName="util"
Sep 17 18:06:17.798811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798823    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93d30ef-7787-42c6-8d0e-8543dd1971fb" containerName="util"
Sep 17 18:06:17.799554 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798828    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0723283-880f-49a5-8324-34094c7af3a0" containerName="pull"
Sep 17 18:06:17.799554 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798832    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0723283-880f-49a5-8324-34094c7af3a0" containerName="pull"
Sep 17 18:06:17.799554 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798881    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0723283-880f-49a5-8324-34094c7af3a0" containerName="extract"
Sep 17 18:06:17.799554 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798888    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="75d15832-833f-49e8-9cab-914b54b3b914" containerName="extract"
Sep 17 18:06:17.799554 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798894    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93d30ef-7787-42c6-8d0e-8543dd1971fb" containerName="extract"
Sep 17 18:06:17.799554 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.798898    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47a502c-2108-4f57-a7f2-90d7af11b5bd" containerName="extract"
Sep 17 18:06:17.803498 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.803479    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-5sq9k"
Sep 17 18:06:17.806553 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.806507    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 18:06:17.807149 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.807127    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-57t9t\""
Sep 17 18:06:17.807453 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.807436    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 18:06:17.815929 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.815907    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-5sq9k"]
Sep 17 18:06:17.934294 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.934261    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:17.949372 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:17.949349    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt92k\" (UniqueName: \"kubernetes.io/projected/bb91b6c1-b60e-40d8-a953-a0c3bc438381-kube-api-access-xt92k\") pod \"authorino-operator-6fc8f864f4-5sq9k\" (UID: \"bb91b6c1-b60e-40d8-a953-a0c3bc438381\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-5sq9k"
Sep 17 18:06:18.034744 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:18.034714    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:18.049888 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:18.049837    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt92k\" (UniqueName: \"kubernetes.io/projected/bb91b6c1-b60e-40d8-a953-a0c3bc438381-kube-api-access-xt92k\") pod \"authorino-operator-6fc8f864f4-5sq9k\" (UID: \"bb91b6c1-b60e-40d8-a953-a0c3bc438381\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-5sq9k"
Sep 17 18:06:18.062255 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:18.062235    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt92k\" (UniqueName: \"kubernetes.io/projected/bb91b6c1-b60e-40d8-a953-a0c3bc438381-kube-api-access-xt92k\") pod \"authorino-operator-6fc8f864f4-5sq9k\" (UID: \"bb91b6c1-b60e-40d8-a953-a0c3bc438381\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-5sq9k"
Sep 17 18:06:18.113172 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:18.113137    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-5sq9k"
Sep 17 18:06:18.385623 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:18.385599    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-5sq9k"]
Sep 17 18:06:18.387688 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:06:18.387664    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb91b6c1_b60e_40d8_a953_a0c3bc438381.slice/crio-044884b4b1f7d57eda8cbfc2030cbf5b9bbfceca37f3956b87c2284e083a3ac7 WatchSource:0}: Error finding container 044884b4b1f7d57eda8cbfc2030cbf5b9bbfceca37f3956b87c2284e083a3ac7: Status 404 returned error can't find the container with id 044884b4b1f7d57eda8cbfc2030cbf5b9bbfceca37f3956b87c2284e083a3ac7
Sep 17 18:06:18.403704 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:18.403672    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-5sq9k" event={"ID":"bb91b6c1-b60e-40d8-a953-a0c3bc438381","Type":"ContainerStarted","Data":"044884b4b1f7d57eda8cbfc2030cbf5b9bbfceca37f3956b87c2284e083a3ac7"}
Sep 17 18:06:18.695247 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:18.695171    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:19.351359 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:19.351322    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:20.415749 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:20.415713    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-5sq9k" event={"ID":"bb91b6c1-b60e-40d8-a953-a0c3bc438381","Type":"ContainerStarted","Data":"577fff06c27b93731d58a62ee3ef6c78b48e7cf7fe8ba879b3a165ebd5d541c5"}
Sep 17 18:06:20.416116 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:20.415774    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-5sq9k"
Sep 17 18:06:20.449214 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:20.449155    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-5sq9k" podStartSLOduration=1.987044174 podStartE2EDuration="3.449138898s" totalImagesPullingTime="1.462094724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:20.447274268 +0000 UTC m=+550.748096593" watchObservedRunningTime="2026-09-17 18:06:20.449138898 +0000 UTC m=+550.749961223"
Sep 17 18:06:20.926537 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:20.926482    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:21.064792 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:21.064751    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:21.289397 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:21.289360    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:21.391726 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:21.391693    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:21.687780 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:21.687703    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:21.951911 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:21.951835    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:25.094467 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:25.094437    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:25.368394 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:25.368311    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:26.832939 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:26.832906    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8j28v"]
Sep 17 18:06:26.835318 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:26.835301    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8j28v"
Sep 17 18:06:26.840215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:26.840193    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hq587\""
Sep 17 18:06:26.849996 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:26.849975    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8j28v"]
Sep 17 18:06:26.911811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:26.911782    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkddv\" (UniqueName: \"kubernetes.io/projected/288ad38b-180c-4a5c-83bd-ad3b309d3c62-kube-api-access-bkddv\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-8j28v\" (UID: \"288ad38b-180c-4a5c-83bd-ad3b309d3c62\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8j28v"
Sep 17 18:06:27.013281 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:27.013224    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkddv\" (UniqueName: \"kubernetes.io/projected/288ad38b-180c-4a5c-83bd-ad3b309d3c62-kube-api-access-bkddv\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-8j28v\" (UID: \"288ad38b-180c-4a5c-83bd-ad3b309d3c62\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8j28v"
Sep 17 18:06:27.022502 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:27.022472    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkddv\" (UniqueName: \"kubernetes.io/projected/288ad38b-180c-4a5c-83bd-ad3b309d3c62-kube-api-access-bkddv\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-8j28v\" (UID: \"288ad38b-180c-4a5c-83bd-ad3b309d3c62\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8j28v"
Sep 17 18:06:27.144900 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:27.144811    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8j28v"
Sep 17 18:06:27.427616 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:27.427582    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8j28v"]
Sep 17 18:06:27.428358 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:06:27.428327    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288ad38b_180c_4a5c_83bd_ad3b309d3c62.slice/crio-c3e904267afccbbbb6d10e4d3e2c942cddf70287f6d0e712570986c9a7bf9a69 WatchSource:0}: Error finding container c3e904267afccbbbb6d10e4d3e2c942cddf70287f6d0e712570986c9a7bf9a69: Status 404 returned error can't find the container with id c3e904267afccbbbb6d10e4d3e2c942cddf70287f6d0e712570986c9a7bf9a69
Sep 17 18:06:27.450699 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:27.450671    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8j28v" event={"ID":"288ad38b-180c-4a5c-83bd-ad3b309d3c62","Type":"ContainerStarted","Data":"c3e904267afccbbbb6d10e4d3e2c942cddf70287f6d0e712570986c9a7bf9a69"}
Sep 17 18:06:28.119082 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:28.119047    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:28.241484 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:28.241440    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:28.346385 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:28.346353    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:28.454816 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:28.454723    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:29.461892 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:29.461858    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8j28v" event={"ID":"288ad38b-180c-4a5c-83bd-ad3b309d3c62","Type":"ContainerStarted","Data":"4f649e5dc87acff24b5826691ca729e738e36f0a5cc7feb6348f2073edcae110"}
Sep 17 18:06:29.462276 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:29.461942    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8j28v"
Sep 17 18:06:29.485146 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:29.485093    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8j28v" podStartSLOduration=1.5340475150000001 podStartE2EDuration="3.48507583s" totalImagesPullingTime="1.951028315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:29.484620608 +0000 UTC m=+559.785442936" watchObservedRunningTime="2026-09-17 18:06:29.48507583 +0000 UTC m=+559.785898153"
Sep 17 18:06:31.422890 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:31.422854    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-5sq9k"
Sep 17 18:06:31.836088 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:31.836054    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:31.932130 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:31.932071    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:32.056041 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:32.055996    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:32.232841 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:32.232742    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:34.675887 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:34.675851    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:36.289020 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:36.288984    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:37.822405 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:37.822375    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:37.928698 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:37.928666    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:40.284650 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:40.284619    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:40.469618 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:40.469577    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8j28v"
Sep 17 18:06:44.749193 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:44.749115    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:44.867988 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:44.867953    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:47.990459 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:47.990429    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:48.094117 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:48.094062    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:48.954689 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:48.954661    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:49.064912 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:49.064879    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:51.390298 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:51.390265    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:51.509432 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:51.509388    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:57.553058 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:57.553006    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:57.653139 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:57.653112    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:57.761123 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:57.761092    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:58.107747 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:58.107680    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:58.196024 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:06:58.195991    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:04.625376 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:04.625324    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:04.803633 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:04.803603    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:11.772403 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:11.772374    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:07:11.774275 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:11.774253    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:07:17.433857 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:17.433824    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:17.525533 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:17.525487    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:17.627480 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:17.627441    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:17.826790 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:17.826760    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6g482"]
Sep 17 18:07:17.830086 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:17.830065    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-6g482"
Sep 17 18:07:17.833927 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:17.833904    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 18:07:17.834014 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:17.833958    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rvg55\""
Sep 17 18:07:17.840566 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:17.840541    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6g482"]
Sep 17 18:07:17.886933 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:17.886901    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6g482"]
Sep 17 18:07:17.902736 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:17.902700    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da-config-file\") pod \"limitador-limitador-84994f7ffc-6g482\" (UID: \"1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6g482"
Sep 17 18:07:17.902736 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:17.902732    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlnhw\" (UniqueName: \"kubernetes.io/projected/1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da-kube-api-access-jlnhw\") pod \"limitador-limitador-84994f7ffc-6g482\" (UID: \"1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6g482"
Sep 17 18:07:18.003330 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:18.003286    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da-config-file\") pod \"limitador-limitador-84994f7ffc-6g482\" (UID: \"1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6g482"
Sep 17 18:07:18.003555 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:18.003336    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlnhw\" (UniqueName: \"kubernetes.io/projected/1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da-kube-api-access-jlnhw\") pod \"limitador-limitador-84994f7ffc-6g482\" (UID: \"1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6g482"
Sep 17 18:07:18.003987 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:18.003967    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da-config-file\") pod \"limitador-limitador-84994f7ffc-6g482\" (UID: \"1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6g482"
Sep 17 18:07:18.015283 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:18.015256    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlnhw\" (UniqueName: \"kubernetes.io/projected/1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da-kube-api-access-jlnhw\") pod \"limitador-limitador-84994f7ffc-6g482\" (UID: \"1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6g482"
Sep 17 18:07:18.140402 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:18.140314    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-6g482"
Sep 17 18:07:18.350422 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:18.350388    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:18.414317 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:18.414289    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6g482"]
Sep 17 18:07:18.416853 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:07:18.416822    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b57ec3f_030a_47e0_9bf3_ac3e1fafe0da.slice/crio-4317f41b60c4312f7ba2014f5f45fdfc71a01d6126775d8cea1476c3d82a2968 WatchSource:0}: Error finding container 4317f41b60c4312f7ba2014f5f45fdfc71a01d6126775d8cea1476c3d82a2968: Status 404 returned error can't find the container with id 4317f41b60c4312f7ba2014f5f45fdfc71a01d6126775d8cea1476c3d82a2968
Sep 17 18:07:18.474593 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:18.474562    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:18.709324 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:18.709238    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-6g482" event={"ID":"1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da","Type":"ContainerStarted","Data":"4317f41b60c4312f7ba2014f5f45fdfc71a01d6126775d8cea1476c3d82a2968"}
Sep 17 18:07:20.721777 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:20.721692    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-6g482" event={"ID":"1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da","Type":"ContainerStarted","Data":"7a6000d567070d4db50a79d7f8b83b0b3a6e7d9afa887458e1c7bcd55edb46a6"}
Sep 17 18:07:20.721777 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:20.721762    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-6g482"
Sep 17 18:07:20.738693 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:20.738600    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-6g482" podStartSLOduration=2.181806759 podStartE2EDuration="3.738585193s" totalImagesPullingTime="1.556778434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:20.738292064 +0000 UTC m=+611.039114388" watchObservedRunningTime="2026-09-17 18:07:20.738585193 +0000 UTC m=+611.039407510"
Sep 17 18:07:31.727424 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:31.727389    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-6g482"
Sep 17 18:07:37.967303 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:37.967268    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:38.153702 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:38.153666    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:45.430080 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:45.430044    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:46.762376 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:46.762336    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:46.860125 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:46.860094    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:55.623824 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:07:55.623788    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:07.697604 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.697570    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"]
Sep 17 18:08:07.698046 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.697750    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22" podUID="704e2be6-21b0-420a-afb0-06bd98d985ed" containerName="discovery" containerID="cri-o://df3aed60fa51feebfc0b1a6c8061ebcdeda3822ae8942f3a829e212db2ec2701" gracePeriod=30
Sep 17 18:08:07.948126 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.948063    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:08:07.967990 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.967960    2574 generic.go:356] "Generic (PLEG): container finished" podID="704e2be6-21b0-420a-afb0-06bd98d985ed" containerID="df3aed60fa51feebfc0b1a6c8061ebcdeda3822ae8942f3a829e212db2ec2701" exitCode=0
Sep 17 18:08:07.968104 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.968001    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"
Sep 17 18:08:07.968104 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.968039    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22" event={"ID":"704e2be6-21b0-420a-afb0-06bd98d985ed","Type":"ContainerDied","Data":"df3aed60fa51feebfc0b1a6c8061ebcdeda3822ae8942f3a829e212db2ec2701"}
Sep 17 18:08:07.968179 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.968109    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22" event={"ID":"704e2be6-21b0-420a-afb0-06bd98d985ed","Type":"ContainerDied","Data":"8f27d8b2ae12e780ca164b73dfa4cbd40c04c8577f251fb4a8b6618fad9153cb"}
Sep 17 18:08:07.968179 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.968128    2574 scope.go:117] "RemoveContainer" containerID="df3aed60fa51feebfc0b1a6c8061ebcdeda3822ae8942f3a829e212db2ec2701"
Sep 17 18:08:07.978529 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.978486    2574 scope.go:117] "RemoveContainer" containerID="df3aed60fa51feebfc0b1a6c8061ebcdeda3822ae8942f3a829e212db2ec2701"
Sep 17 18:08:07.978833 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:08:07.978813    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3aed60fa51feebfc0b1a6c8061ebcdeda3822ae8942f3a829e212db2ec2701\": container with ID starting with df3aed60fa51feebfc0b1a6c8061ebcdeda3822ae8942f3a829e212db2ec2701 not found: ID does not exist" containerID="df3aed60fa51feebfc0b1a6c8061ebcdeda3822ae8942f3a829e212db2ec2701"
Sep 17 18:08:07.978894 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.978844    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3aed60fa51feebfc0b1a6c8061ebcdeda3822ae8942f3a829e212db2ec2701"} err="failed to get container status \"df3aed60fa51feebfc0b1a6c8061ebcdeda3822ae8942f3a829e212db2ec2701\": rpc error: code = NotFound desc = could not find container \"df3aed60fa51feebfc0b1a6c8061ebcdeda3822ae8942f3a829e212db2ec2701\": container with ID starting with df3aed60fa51feebfc0b1a6c8061ebcdeda3822ae8942f3a829e212db2ec2701 not found: ID does not exist"
Sep 17 18:08:07.990699 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.990677    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-cacerts\") pod \"704e2be6-21b0-420a-afb0-06bd98d985ed\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") "
Sep 17 18:08:07.990810 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.990727    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/704e2be6-21b0-420a-afb0-06bd98d985ed-local-certs\") pod \"704e2be6-21b0-420a-afb0-06bd98d985ed\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") "
Sep 17 18:08:07.990914 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.990890    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-kubeconfig\") pod \"704e2be6-21b0-420a-afb0-06bd98d985ed\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") "
Sep 17 18:08:07.991029 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.990943    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-token\") pod \"704e2be6-21b0-420a-afb0-06bd98d985ed\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") "
Sep 17 18:08:07.991029 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.990977    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-csr-ca-configmap\") pod \"704e2be6-21b0-420a-afb0-06bd98d985ed\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") "
Sep 17 18:08:07.991029 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.991010    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhxlz\" (UniqueName: \"kubernetes.io/projected/704e2be6-21b0-420a-afb0-06bd98d985ed-kube-api-access-lhxlz\") pod \"704e2be6-21b0-420a-afb0-06bd98d985ed\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") "
Sep 17 18:08:07.991193 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.991052    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-csr-dns-cert\") pod \"704e2be6-21b0-420a-afb0-06bd98d985ed\" (UID: \"704e2be6-21b0-420a-afb0-06bd98d985ed\") "
Sep 17 18:08:07.992161 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.991635    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "704e2be6-21b0-420a-afb0-06bd98d985ed" (UID: "704e2be6-21b0-420a-afb0-06bd98d985ed"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:07.993651 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.993617    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704e2be6-21b0-420a-afb0-06bd98d985ed-kube-api-access-lhxlz" (OuterVolumeSpecName: "kube-api-access-lhxlz") pod "704e2be6-21b0-420a-afb0-06bd98d985ed" (UID: "704e2be6-21b0-420a-afb0-06bd98d985ed"). InnerVolumeSpecName "kube-api-access-lhxlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:07.993768 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.993671    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704e2be6-21b0-420a-afb0-06bd98d985ed-local-certs" (OuterVolumeSpecName: "local-certs") pod "704e2be6-21b0-420a-afb0-06bd98d985ed" (UID: "704e2be6-21b0-420a-afb0-06bd98d985ed"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:07.993839 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.993791    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-token" (OuterVolumeSpecName: "istio-token") pod "704e2be6-21b0-420a-afb0-06bd98d985ed" (UID: "704e2be6-21b0-420a-afb0-06bd98d985ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:07.993839 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.993811    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "704e2be6-21b0-420a-afb0-06bd98d985ed" (UID: "704e2be6-21b0-420a-afb0-06bd98d985ed"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:07.994046 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.994030    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "704e2be6-21b0-420a-afb0-06bd98d985ed" (UID: "704e2be6-21b0-420a-afb0-06bd98d985ed"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:07.994111 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:07.994088    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-cacerts" (OuterVolumeSpecName: "cacerts") pod "704e2be6-21b0-420a-afb0-06bd98d985ed" (UID: "704e2be6-21b0-420a-afb0-06bd98d985ed"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:08.091822 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:08.091784    2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-cacerts\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:08:08.091822 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:08.091818    2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/704e2be6-21b0-420a-afb0-06bd98d985ed-local-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:08:08.091822 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:08.091828    2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-kubeconfig\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:08:08.091822 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:08.091836    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-token\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:08:08.092044 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:08.091843    2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-csr-ca-configmap\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:08:08.092044 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:08.091851    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhxlz\" (UniqueName: \"kubernetes.io/projected/704e2be6-21b0-420a-afb0-06bd98d985ed-kube-api-access-lhxlz\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:08:08.092044 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:08.091859    2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/704e2be6-21b0-420a-afb0-06bd98d985ed-istio-csr-dns-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:08:08.289848 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:08.289820    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"]
Sep 17 18:08:08.293194 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:08.293168    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzf22"]
Sep 17 18:08:09.938549 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:09.938498    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="704e2be6-21b0-420a-afb0-06bd98d985ed" path="/var/lib/kubelet/pods/704e2be6-21b0-420a-afb0-06bd98d985ed/volumes"
Sep 17 18:08:14.931315 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:14.931279    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:15.011429 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.011394    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5bf6d75b64-5cwc5"]
Sep 17 18:08:15.011772 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.011759    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704e2be6-21b0-420a-afb0-06bd98d985ed" containerName="discovery"
Sep 17 18:08:15.011822 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.011774    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="704e2be6-21b0-420a-afb0-06bd98d985ed" containerName="discovery"
Sep 17 18:08:15.011857 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.011829    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="704e2be6-21b0-420a-afb0-06bd98d985ed" containerName="discovery"
Sep 17 18:08:15.014904 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.014882    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5"
Sep 17 18:08:15.017561 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.017537    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-bkfxz\""
Sep 17 18:08:15.017680 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.017580    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:08:15.018435 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.018410    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 18:08:15.018435 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.018433    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:08:15.026699 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.026674    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5bf6d75b64-5cwc5"]
Sep 17 18:08:15.029963 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.029937    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-78cb6989d4-6jpqg"]
Sep 17 18:08:15.032233 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.032218    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-78cb6989d4-6jpqg"
Sep 17 18:08:15.034669 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.034646    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 18:08:15.034781 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.034702    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-bkfwg\""
Sep 17 18:08:15.048862 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.048834    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-78cb6989d4-6jpqg"]
Sep 17 18:08:15.053131 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.053105    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:15.156169 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.156128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlr28\" (UniqueName: \"kubernetes.io/projected/a283db81-4ef8-41b8-b283-977dc8bac84e-kube-api-access-jlr28\") pod \"llmisvc-controller-manager-78cb6989d4-6jpqg\" (UID: \"a283db81-4ef8-41b8-b283-977dc8bac84e\") " pod="kserve/llmisvc-controller-manager-78cb6989d4-6jpqg"
Sep 17 18:08:15.156355 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.156188    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6038fb52-2094-4283-9455-5cc2db8cef07-cert\") pod \"kserve-controller-manager-5bf6d75b64-5cwc5\" (UID: \"6038fb52-2094-4283-9455-5cc2db8cef07\") " pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5"
Sep 17 18:08:15.156355 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.156249    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgfvp\" (UniqueName: \"kubernetes.io/projected/6038fb52-2094-4283-9455-5cc2db8cef07-kube-api-access-kgfvp\") pod \"kserve-controller-manager-5bf6d75b64-5cwc5\" (UID: \"6038fb52-2094-4283-9455-5cc2db8cef07\") " pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5"
Sep 17 18:08:15.156355 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.156274    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a283db81-4ef8-41b8-b283-977dc8bac84e-cert\") pod \"llmisvc-controller-manager-78cb6989d4-6jpqg\" (UID: \"a283db81-4ef8-41b8-b283-977dc8bac84e\") " pod="kserve/llmisvc-controller-manager-78cb6989d4-6jpqg"
Sep 17 18:08:15.160547 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.160489    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:15.257629 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.257597    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgfvp\" (UniqueName: \"kubernetes.io/projected/6038fb52-2094-4283-9455-5cc2db8cef07-kube-api-access-kgfvp\") pod \"kserve-controller-manager-5bf6d75b64-5cwc5\" (UID: \"6038fb52-2094-4283-9455-5cc2db8cef07\") " pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5"
Sep 17 18:08:15.257804 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.257635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a283db81-4ef8-41b8-b283-977dc8bac84e-cert\") pod \"llmisvc-controller-manager-78cb6989d4-6jpqg\" (UID: \"a283db81-4ef8-41b8-b283-977dc8bac84e\") " pod="kserve/llmisvc-controller-manager-78cb6989d4-6jpqg"
Sep 17 18:08:15.257804 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.257688    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlr28\" (UniqueName: \"kubernetes.io/projected/a283db81-4ef8-41b8-b283-977dc8bac84e-kube-api-access-jlr28\") pod \"llmisvc-controller-manager-78cb6989d4-6jpqg\" (UID: \"a283db81-4ef8-41b8-b283-977dc8bac84e\") " pod="kserve/llmisvc-controller-manager-78cb6989d4-6jpqg"
Sep 17 18:08:15.257804 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.257718    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6038fb52-2094-4283-9455-5cc2db8cef07-cert\") pod \"kserve-controller-manager-5bf6d75b64-5cwc5\" (UID: \"6038fb52-2094-4283-9455-5cc2db8cef07\") " pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5"
Sep 17 18:08:15.260544 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.260506    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a283db81-4ef8-41b8-b283-977dc8bac84e-cert\") pod \"llmisvc-controller-manager-78cb6989d4-6jpqg\" (UID: \"a283db81-4ef8-41b8-b283-977dc8bac84e\") " pod="kserve/llmisvc-controller-manager-78cb6989d4-6jpqg"
Sep 17 18:08:15.260682 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.260662    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6038fb52-2094-4283-9455-5cc2db8cef07-cert\") pod \"kserve-controller-manager-5bf6d75b64-5cwc5\" (UID: \"6038fb52-2094-4283-9455-5cc2db8cef07\") " pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5"
Sep 17 18:08:15.260870 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.260853    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:15.266630 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.266604    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgfvp\" (UniqueName: \"kubernetes.io/projected/6038fb52-2094-4283-9455-5cc2db8cef07-kube-api-access-kgfvp\") pod \"kserve-controller-manager-5bf6d75b64-5cwc5\" (UID: \"6038fb52-2094-4283-9455-5cc2db8cef07\") " pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5"
Sep 17 18:08:15.266953 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.266923    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlr28\" (UniqueName: \"kubernetes.io/projected/a283db81-4ef8-41b8-b283-977dc8bac84e-kube-api-access-jlr28\") pod \"llmisvc-controller-manager-78cb6989d4-6jpqg\" (UID: \"a283db81-4ef8-41b8-b283-977dc8bac84e\") " pod="kserve/llmisvc-controller-manager-78cb6989d4-6jpqg"
Sep 17 18:08:15.325577 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.325539    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5"
Sep 17 18:08:15.342169 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.342142    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-78cb6989d4-6jpqg"
Sep 17 18:08:15.598153 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.598125    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5bf6d75b64-5cwc5"]
Sep 17 18:08:15.600131 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:08:15.600100    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6038fb52_2094_4283_9455_5cc2db8cef07.slice/crio-60ee4de13215ad6deac8f7d993d04dd9fa9b6e413a7fdaef789fdb416d4e0b2f WatchSource:0}: Error finding container 60ee4de13215ad6deac8f7d993d04dd9fa9b6e413a7fdaef789fdb416d4e0b2f: Status 404 returned error can't find the container with id 60ee4de13215ad6deac8f7d993d04dd9fa9b6e413a7fdaef789fdb416d4e0b2f
Sep 17 18:08:15.601561 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.601542    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:08:15.632918 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.632892    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-78cb6989d4-6jpqg"]
Sep 17 18:08:15.634222 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:08:15.634195    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda283db81_4ef8_41b8_b283_977dc8bac84e.slice/crio-c15a274959385cbea27b44012b1e5652134085f9d98cf28dbf64b377963735d9 WatchSource:0}: Error finding container c15a274959385cbea27b44012b1e5652134085f9d98cf28dbf64b377963735d9: Status 404 returned error can't find the container with id c15a274959385cbea27b44012b1e5652134085f9d98cf28dbf64b377963735d9
Sep 17 18:08:15.663916 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.663888    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:15.878068 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:15.877978    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:16.013694 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:16.013652    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5" event={"ID":"6038fb52-2094-4283-9455-5cc2db8cef07","Type":"ContainerStarted","Data":"60ee4de13215ad6deac8f7d993d04dd9fa9b6e413a7fdaef789fdb416d4e0b2f"}
Sep 17 18:08:16.014963 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:16.014925    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-78cb6989d4-6jpqg" event={"ID":"a283db81-4ef8-41b8-b283-977dc8bac84e","Type":"ContainerStarted","Data":"c15a274959385cbea27b44012b1e5652134085f9d98cf28dbf64b377963735d9"}
Sep 17 18:08:19.035315 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:19.035279    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-78cb6989d4-6jpqg" event={"ID":"a283db81-4ef8-41b8-b283-977dc8bac84e","Type":"ContainerStarted","Data":"4f4d43f3e6b1f0ff5f31bb23f96736fc15d19ac0209c3dec9e6a58c149695d15"}
Sep 17 18:08:19.035740 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:19.035443    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-78cb6989d4-6jpqg"
Sep 17 18:08:19.036915 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:19.036891    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5" event={"ID":"6038fb52-2094-4283-9455-5cc2db8cef07","Type":"ContainerStarted","Data":"d3b6948c504dd6dab07627017774a8a234c4b11d4980dad2ff6039e753fa1838"}
Sep 17 18:08:19.037026 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:19.036953    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5"
Sep 17 18:08:19.056998 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:19.056957    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-78cb6989d4-6jpqg" podStartSLOduration=0.782863143 podStartE2EDuration="4.056944063s" totalImagesPullingTime="3.27408092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:19.055295776 +0000 UTC m=+669.356118099" watchObservedRunningTime="2026-09-17 18:08:19.056944063 +0000 UTC m=+669.357766386"
Sep 17 18:08:19.080405 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:19.080360    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5" podStartSLOduration=1.7757153639999999 podStartE2EDuration="5.080347236s" totalImagesPullingTime="3.304631872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:19.078279927 +0000 UTC m=+669.379102254" watchObservedRunningTime="2026-09-17 18:08:19.080347236 +0000 UTC m=+669.381169559"
Sep 17 18:08:50.043985 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:50.043950    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-78cb6989d4-6jpqg"
Sep 17 18:08:50.046958 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:50.046931    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5"
Sep 17 18:08:58.318823 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.318744    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5bf6d75b64-5cwc5"]
Sep 17 18:08:58.319323 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.318939    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5" podUID="6038fb52-2094-4283-9455-5cc2db8cef07" containerName="manager" containerID="cri-o://d3b6948c504dd6dab07627017774a8a234c4b11d4980dad2ff6039e753fa1838" gracePeriod=10
Sep 17 18:08:58.343685 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.343654    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5bf6d75b64-5nkxk"]
Sep 17 18:08:58.345955 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.345937    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5bf6d75b64-5nkxk"
Sep 17 18:08:58.358174 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.358149    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5bf6d75b64-5nkxk"]
Sep 17 18:08:58.412720 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.412685    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2115c54c-5324-47af-8226-5c429bf0e171-cert\") pod \"kserve-controller-manager-5bf6d75b64-5nkxk\" (UID: \"2115c54c-5324-47af-8226-5c429bf0e171\") " pod="kserve/kserve-controller-manager-5bf6d75b64-5nkxk"
Sep 17 18:08:58.412895 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.412728    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nnxs\" (UniqueName: \"kubernetes.io/projected/2115c54c-5324-47af-8226-5c429bf0e171-kube-api-access-2nnxs\") pod \"kserve-controller-manager-5bf6d75b64-5nkxk\" (UID: \"2115c54c-5324-47af-8226-5c429bf0e171\") " pod="kserve/kserve-controller-manager-5bf6d75b64-5nkxk"
Sep 17 18:08:58.514084 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.514049    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2115c54c-5324-47af-8226-5c429bf0e171-cert\") pod \"kserve-controller-manager-5bf6d75b64-5nkxk\" (UID: \"2115c54c-5324-47af-8226-5c429bf0e171\") " pod="kserve/kserve-controller-manager-5bf6d75b64-5nkxk"
Sep 17 18:08:58.514084 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.514093    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nnxs\" (UniqueName: \"kubernetes.io/projected/2115c54c-5324-47af-8226-5c429bf0e171-kube-api-access-2nnxs\") pod \"kserve-controller-manager-5bf6d75b64-5nkxk\" (UID: \"2115c54c-5324-47af-8226-5c429bf0e171\") " pod="kserve/kserve-controller-manager-5bf6d75b64-5nkxk"
Sep 17 18:08:58.516731 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.516702    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2115c54c-5324-47af-8226-5c429bf0e171-cert\") pod \"kserve-controller-manager-5bf6d75b64-5nkxk\" (UID: \"2115c54c-5324-47af-8226-5c429bf0e171\") " pod="kserve/kserve-controller-manager-5bf6d75b64-5nkxk"
Sep 17 18:08:58.522360 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.522333    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nnxs\" (UniqueName: \"kubernetes.io/projected/2115c54c-5324-47af-8226-5c429bf0e171-kube-api-access-2nnxs\") pod \"kserve-controller-manager-5bf6d75b64-5nkxk\" (UID: \"2115c54c-5324-47af-8226-5c429bf0e171\") " pod="kserve/kserve-controller-manager-5bf6d75b64-5nkxk"
Sep 17 18:08:58.553347 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.553326    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5"
Sep 17 18:08:58.615459 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.615384    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgfvp\" (UniqueName: \"kubernetes.io/projected/6038fb52-2094-4283-9455-5cc2db8cef07-kube-api-access-kgfvp\") pod \"6038fb52-2094-4283-9455-5cc2db8cef07\" (UID: \"6038fb52-2094-4283-9455-5cc2db8cef07\") "
Sep 17 18:08:58.615459 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.615440    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6038fb52-2094-4283-9455-5cc2db8cef07-cert\") pod \"6038fb52-2094-4283-9455-5cc2db8cef07\" (UID: \"6038fb52-2094-4283-9455-5cc2db8cef07\") "
Sep 17 18:08:58.617713 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.617680    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6038fb52-2094-4283-9455-5cc2db8cef07-cert" (OuterVolumeSpecName: "cert") pod "6038fb52-2094-4283-9455-5cc2db8cef07" (UID: "6038fb52-2094-4283-9455-5cc2db8cef07"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:58.617713 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.617702    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6038fb52-2094-4283-9455-5cc2db8cef07-kube-api-access-kgfvp" (OuterVolumeSpecName: "kube-api-access-kgfvp") pod "6038fb52-2094-4283-9455-5cc2db8cef07" (UID: "6038fb52-2094-4283-9455-5cc2db8cef07"). InnerVolumeSpecName "kube-api-access-kgfvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:58.699834 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.699792    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5bf6d75b64-5nkxk"
Sep 17 18:08:58.716395 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.716360    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgfvp\" (UniqueName: \"kubernetes.io/projected/6038fb52-2094-4283-9455-5cc2db8cef07-kube-api-access-kgfvp\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:08:58.716542 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.716398    2574 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6038fb52-2094-4283-9455-5cc2db8cef07-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:08:58.970703 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:58.970675    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5bf6d75b64-5nkxk"]
Sep 17 18:08:58.972532 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:08:58.972481    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2115c54c_5324_47af_8226_5c429bf0e171.slice/crio-5340e781a5e61f6cab704c419997e70c1bbfec483e5919076b22b4684c0dcdbb WatchSource:0}: Error finding container 5340e781a5e61f6cab704c419997e70c1bbfec483e5919076b22b4684c0dcdbb: Status 404 returned error can't find the container with id 5340e781a5e61f6cab704c419997e70c1bbfec483e5919076b22b4684c0dcdbb
Sep 17 18:08:59.239043 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:59.238952    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5bf6d75b64-5nkxk" event={"ID":"2115c54c-5324-47af-8226-5c429bf0e171","Type":"ContainerStarted","Data":"5340e781a5e61f6cab704c419997e70c1bbfec483e5919076b22b4684c0dcdbb"}
Sep 17 18:08:59.240101 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:59.240072    2574 generic.go:356] "Generic (PLEG): container finished" podID="6038fb52-2094-4283-9455-5cc2db8cef07" containerID="d3b6948c504dd6dab07627017774a8a234c4b11d4980dad2ff6039e753fa1838" exitCode=0
Sep 17 18:08:59.240225 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:59.240107    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5" event={"ID":"6038fb52-2094-4283-9455-5cc2db8cef07","Type":"ContainerDied","Data":"d3b6948c504dd6dab07627017774a8a234c4b11d4980dad2ff6039e753fa1838"}
Sep 17 18:08:59.240225 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:59.240113    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5"
Sep 17 18:08:59.240225 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:59.240141    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5bf6d75b64-5cwc5" event={"ID":"6038fb52-2094-4283-9455-5cc2db8cef07","Type":"ContainerDied","Data":"60ee4de13215ad6deac8f7d993d04dd9fa9b6e413a7fdaef789fdb416d4e0b2f"}
Sep 17 18:08:59.240225 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:59.240164    2574 scope.go:117] "RemoveContainer" containerID="d3b6948c504dd6dab07627017774a8a234c4b11d4980dad2ff6039e753fa1838"
Sep 17 18:08:59.249024 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:59.249007    2574 scope.go:117] "RemoveContainer" containerID="d3b6948c504dd6dab07627017774a8a234c4b11d4980dad2ff6039e753fa1838"
Sep 17 18:08:59.249298 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:08:59.249280    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b6948c504dd6dab07627017774a8a234c4b11d4980dad2ff6039e753fa1838\": container with ID starting with d3b6948c504dd6dab07627017774a8a234c4b11d4980dad2ff6039e753fa1838 not found: ID does not exist" containerID="d3b6948c504dd6dab07627017774a8a234c4b11d4980dad2ff6039e753fa1838"
Sep 17 18:08:59.249370 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:59.249307    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b6948c504dd6dab07627017774a8a234c4b11d4980dad2ff6039e753fa1838"} err="failed to get container status \"d3b6948c504dd6dab07627017774a8a234c4b11d4980dad2ff6039e753fa1838\": rpc error: code = NotFound desc = could not find container \"d3b6948c504dd6dab07627017774a8a234c4b11d4980dad2ff6039e753fa1838\": container with ID starting with d3b6948c504dd6dab07627017774a8a234c4b11d4980dad2ff6039e753fa1838 not found: ID does not exist"
Sep 17 18:08:59.270219 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:59.270192    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5bf6d75b64-5cwc5"]
Sep 17 18:08:59.281083 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:59.281052    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-5bf6d75b64-5cwc5"]
Sep 17 18:08:59.938416 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:08:59.938385    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6038fb52-2094-4283-9455-5cc2db8cef07" path="/var/lib/kubelet/pods/6038fb52-2094-4283-9455-5cc2db8cef07/volumes"
Sep 17 18:09:00.247655 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:00.247570    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5bf6d75b64-5nkxk" event={"ID":"2115c54c-5324-47af-8226-5c429bf0e171","Type":"ContainerStarted","Data":"e40f8c63be87ea3f3c2997a3defa17a8187e41666816b514ff7b289267f8fb7a"}
Sep 17 18:09:00.247798 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:00.247698    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5bf6d75b64-5nkxk"
Sep 17 18:09:00.266861 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:00.266810    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5bf6d75b64-5nkxk" podStartSLOduration=1.908718027 podStartE2EDuration="2.266793726s" totalImagesPullingTime="358.075699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:00.264654458 +0000 UTC m=+710.565476787" watchObservedRunningTime="2026-09-17 18:09:00.266793726 +0000 UTC m=+710.567616050"
Sep 17 18:09:31.258527 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:31.258479    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5bf6d75b64-5nkxk"
Sep 17 18:09:32.525702 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.525671    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:32.561940 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.561907    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-pp67m"]
Sep 17 18:09:32.562422 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.562407    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6038fb52-2094-4283-9455-5cc2db8cef07" containerName="manager"
Sep 17 18:09:32.562480 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.562426    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6038fb52-2094-4283-9455-5cc2db8cef07" containerName="manager"
Sep 17 18:09:32.562544 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.562533    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6038fb52-2094-4283-9455-5cc2db8cef07" containerName="manager"
Sep 17 18:09:32.566096 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.566078    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-pp67m"
Sep 17 18:09:32.569398 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.569378    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 18:09:32.569398 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.569380    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-tsf8s\""
Sep 17 18:09:32.573415 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.572901    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-6z4kj"]
Sep 17 18:09:32.576499 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.576478    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-pp67m"]
Sep 17 18:09:32.576617 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.576579    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:32.578770 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.578751    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 18:09:32.578855 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.578778    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 18:09:32.578980 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.578965    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-zwr4t\""
Sep 17 18:09:32.584969 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.584949    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-6z4kj"]
Sep 17 18:09:32.606432 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.606401    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px5bb\" (UniqueName: \"kubernetes.io/projected/45c015c7-c631-4d00-b9ae-e812f83dc603-kube-api-access-px5bb\") pod \"odh-model-controller-7b966d7984-6z4kj\" (UID: \"45c015c7-c631-4d00-b9ae-e812f83dc603\") " pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:32.606566 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.606463    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45c015c7-c631-4d00-b9ae-e812f83dc603-cert\") pod \"odh-model-controller-7b966d7984-6z4kj\" (UID: \"45c015c7-c631-4d00-b9ae-e812f83dc603\") " pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:32.606566 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.606505    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45c015c7-c631-4d00-b9ae-e812f83dc603-metrics-certs\") pod \"odh-model-controller-7b966d7984-6z4kj\" (UID: \"45c015c7-c631-4d00-b9ae-e812f83dc603\") " pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:32.606860 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.606621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8295cf9a-e616-47b9-9830-5ad262627455-tls-certs\") pod \"model-serving-api-7b954ffc95-pp67m\" (UID: \"8295cf9a-e616-47b9-9830-5ad262627455\") " pod="kserve/model-serving-api-7b954ffc95-pp67m"
Sep 17 18:09:32.606860 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.606651    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smczv\" (UniqueName: \"kubernetes.io/projected/8295cf9a-e616-47b9-9830-5ad262627455-kube-api-access-smczv\") pod \"model-serving-api-7b954ffc95-pp67m\" (UID: \"8295cf9a-e616-47b9-9830-5ad262627455\") " pod="kserve/model-serving-api-7b954ffc95-pp67m"
Sep 17 18:09:32.707052 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.707021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8295cf9a-e616-47b9-9830-5ad262627455-tls-certs\") pod \"model-serving-api-7b954ffc95-pp67m\" (UID: \"8295cf9a-e616-47b9-9830-5ad262627455\") " pod="kserve/model-serving-api-7b954ffc95-pp67m"
Sep 17 18:09:32.707197 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.707057    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smczv\" (UniqueName: \"kubernetes.io/projected/8295cf9a-e616-47b9-9830-5ad262627455-kube-api-access-smczv\") pod \"model-serving-api-7b954ffc95-pp67m\" (UID: \"8295cf9a-e616-47b9-9830-5ad262627455\") " pod="kserve/model-serving-api-7b954ffc95-pp67m"
Sep 17 18:09:32.707197 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.707106    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px5bb\" (UniqueName: \"kubernetes.io/projected/45c015c7-c631-4d00-b9ae-e812f83dc603-kube-api-access-px5bb\") pod \"odh-model-controller-7b966d7984-6z4kj\" (UID: \"45c015c7-c631-4d00-b9ae-e812f83dc603\") " pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:32.707197 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.707141    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45c015c7-c631-4d00-b9ae-e812f83dc603-cert\") pod \"odh-model-controller-7b966d7984-6z4kj\" (UID: \"45c015c7-c631-4d00-b9ae-e812f83dc603\") " pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:32.707197 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.707170    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45c015c7-c631-4d00-b9ae-e812f83dc603-metrics-certs\") pod \"odh-model-controller-7b966d7984-6z4kj\" (UID: \"45c015c7-c631-4d00-b9ae-e812f83dc603\") " pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:32.707363 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:09:32.707265    2574 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 18:09:32.707363 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:09:32.707276    2574 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 17 18:09:32.707363 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:09:32.707328    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45c015c7-c631-4d00-b9ae-e812f83dc603-cert podName:45c015c7-c631-4d00-b9ae-e812f83dc603 nodeName:}" failed. No retries permitted until 2026-09-17 18:09:33.207313288 +0000 UTC m=+743.508135590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/45c015c7-c631-4d00-b9ae-e812f83dc603-cert") pod "odh-model-controller-7b966d7984-6z4kj" (UID: "45c015c7-c631-4d00-b9ae-e812f83dc603") : secret "odh-model-controller-webhook-cert" not found
Sep 17 18:09:32.707363 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:09:32.707345    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45c015c7-c631-4d00-b9ae-e812f83dc603-metrics-certs podName:45c015c7-c631-4d00-b9ae-e812f83dc603 nodeName:}" failed. No retries permitted until 2026-09-17 18:09:33.207337909 +0000 UTC m=+743.508160211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45c015c7-c631-4d00-b9ae-e812f83dc603-metrics-certs") pod "odh-model-controller-7b966d7984-6z4kj" (UID: "45c015c7-c631-4d00-b9ae-e812f83dc603") : secret "odh-model-controller-metrics-tls" not found
Sep 17 18:09:32.709836 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.709811    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8295cf9a-e616-47b9-9830-5ad262627455-tls-certs\") pod \"model-serving-api-7b954ffc95-pp67m\" (UID: \"8295cf9a-e616-47b9-9830-5ad262627455\") " pod="kserve/model-serving-api-7b954ffc95-pp67m"
Sep 17 18:09:32.717938 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.717912    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px5bb\" (UniqueName: \"kubernetes.io/projected/45c015c7-c631-4d00-b9ae-e812f83dc603-kube-api-access-px5bb\") pod \"odh-model-controller-7b966d7984-6z4kj\" (UID: \"45c015c7-c631-4d00-b9ae-e812f83dc603\") " pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:32.718569 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.718550    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smczv\" (UniqueName: \"kubernetes.io/projected/8295cf9a-e616-47b9-9830-5ad262627455-kube-api-access-smczv\") pod \"model-serving-api-7b954ffc95-pp67m\" (UID: \"8295cf9a-e616-47b9-9830-5ad262627455\") " pod="kserve/model-serving-api-7b954ffc95-pp67m"
Sep 17 18:09:32.877280 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:32.877198    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-pp67m"
Sep 17 18:09:33.151591 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:33.151564    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-pp67m"]
Sep 17 18:09:33.153355 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:09:33.153332    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8295cf9a_e616_47b9_9830_5ad262627455.slice/crio-b2a9e416f59e65ebd18e3cc13509081aed18083e72996be29774102eb07f2f02 WatchSource:0}: Error finding container b2a9e416f59e65ebd18e3cc13509081aed18083e72996be29774102eb07f2f02: Status 404 returned error can't find the container with id b2a9e416f59e65ebd18e3cc13509081aed18083e72996be29774102eb07f2f02
Sep 17 18:09:33.210834 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:33.210804    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45c015c7-c631-4d00-b9ae-e812f83dc603-cert\") pod \"odh-model-controller-7b966d7984-6z4kj\" (UID: \"45c015c7-c631-4d00-b9ae-e812f83dc603\") " pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:33.210834 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:33.210842    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45c015c7-c631-4d00-b9ae-e812f83dc603-metrics-certs\") pod \"odh-model-controller-7b966d7984-6z4kj\" (UID: \"45c015c7-c631-4d00-b9ae-e812f83dc603\") " pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:33.213383 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:33.213352    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45c015c7-c631-4d00-b9ae-e812f83dc603-metrics-certs\") pod \"odh-model-controller-7b966d7984-6z4kj\" (UID: \"45c015c7-c631-4d00-b9ae-e812f83dc603\") " pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:33.213383 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:33.213375    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45c015c7-c631-4d00-b9ae-e812f83dc603-cert\") pod \"odh-model-controller-7b966d7984-6z4kj\" (UID: \"45c015c7-c631-4d00-b9ae-e812f83dc603\") " pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:33.418864 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:33.418778    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-pp67m" event={"ID":"8295cf9a-e616-47b9-9830-5ad262627455","Type":"ContainerStarted","Data":"b2a9e416f59e65ebd18e3cc13509081aed18083e72996be29774102eb07f2f02"}
Sep 17 18:09:33.488152 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:33.488111    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:33.780887 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:33.780858    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-6z4kj"]
Sep 17 18:09:33.782901 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:09:33.782872    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c015c7_c631_4d00_b9ae_e812f83dc603.slice/crio-2066ef947982db6d77ba3f6c2ef9b4f29ce6b2597b3572f11a24e0e2fa8afee3 WatchSource:0}: Error finding container 2066ef947982db6d77ba3f6c2ef9b4f29ce6b2597b3572f11a24e0e2fa8afee3: Status 404 returned error can't find the container with id 2066ef947982db6d77ba3f6c2ef9b4f29ce6b2597b3572f11a24e0e2fa8afee3
Sep 17 18:09:34.426205 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:34.426143    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-6z4kj" event={"ID":"45c015c7-c631-4d00-b9ae-e812f83dc603","Type":"ContainerStarted","Data":"2066ef947982db6d77ba3f6c2ef9b4f29ce6b2597b3572f11a24e0e2fa8afee3"}
Sep 17 18:09:34.718330 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:34.718209    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:37.447998 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:37.447958    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-pp67m" event={"ID":"8295cf9a-e616-47b9-9830-5ad262627455","Type":"ContainerStarted","Data":"4db800bac72d2a283d44e195fc6ecfa099f7057a252029fe50e43c387487dc6c"}
Sep 17 18:09:37.448446 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:37.448059    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-pp67m"
Sep 17 18:09:37.449714 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:37.449686    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-6z4kj" event={"ID":"45c015c7-c631-4d00-b9ae-e812f83dc603","Type":"ContainerStarted","Data":"1deefa4e55b8eea10de4ab2202d09dc42ac8633ffef197b4383a2c4dfa9d7a1f"}
Sep 17 18:09:37.449827 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:37.449766    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:37.468994 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:37.468947    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-pp67m" podStartSLOduration=1.797849973 podStartE2EDuration="5.468933068s" totalImagesPullingTime="3.671083095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:37.466853538 +0000 UTC m=+747.767675861" watchObservedRunningTime="2026-09-17 18:09:37.468933068 +0000 UTC m=+747.769755392"
Sep 17 18:09:48.456778 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:48.456752    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-6z4kj"
Sep 17 18:09:48.458426 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:48.458405    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-pp67m"
Sep 17 18:09:48.475407 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:48.475367    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-6z4kj" podStartSLOduration=13.384126155 podStartE2EDuration="16.475354208s" totalImagesPullingTime="3.091228053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:37.487015739 +0000 UTC m=+747.787838063" watchObservedRunningTime="2026-09-17 18:09:48.475354208 +0000 UTC m=+758.776176532"
Sep 17 18:09:49.283957 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:49.283926    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-gm6lj"]
Sep 17 18:09:49.287566 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:49.287551    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gm6lj"
Sep 17 18:09:49.290019 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:49.289997    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-286dp\""
Sep 17 18:09:49.290019 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:49.290004    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:09:49.290225 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:49.290035    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 18:09:49.294392 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:49.294371    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-gm6lj"]
Sep 17 18:09:49.349708 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:49.349674    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-576xw\" (UniqueName: \"kubernetes.io/projected/5480db01-47da-46d6-90d3-458cacdac16c-kube-api-access-576xw\") pod \"s3-init-gm6lj\" (UID: \"5480db01-47da-46d6-90d3-458cacdac16c\") " pod="kserve/s3-init-gm6lj"
Sep 17 18:09:49.349841 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:49.349722    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5480db01-47da-46d6-90d3-458cacdac16c-model-cache\") pod \"s3-init-gm6lj\" (UID: \"5480db01-47da-46d6-90d3-458cacdac16c\") " pod="kserve/s3-init-gm6lj"
Sep 17 18:09:49.450395 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:49.450367    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576xw\" (UniqueName: \"kubernetes.io/projected/5480db01-47da-46d6-90d3-458cacdac16c-kube-api-access-576xw\") pod \"s3-init-gm6lj\" (UID: \"5480db01-47da-46d6-90d3-458cacdac16c\") " pod="kserve/s3-init-gm6lj"
Sep 17 18:09:49.450555 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:49.450405    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5480db01-47da-46d6-90d3-458cacdac16c-model-cache\") pod \"s3-init-gm6lj\" (UID: \"5480db01-47da-46d6-90d3-458cacdac16c\") " pod="kserve/s3-init-gm6lj"
Sep 17 18:09:49.450786 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:49.450771    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5480db01-47da-46d6-90d3-458cacdac16c-model-cache\") pod \"s3-init-gm6lj\" (UID: \"5480db01-47da-46d6-90d3-458cacdac16c\") " pod="kserve/s3-init-gm6lj"
Sep 17 18:09:49.459364 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:49.459345    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-576xw\" (UniqueName: \"kubernetes.io/projected/5480db01-47da-46d6-90d3-458cacdac16c-kube-api-access-576xw\") pod \"s3-init-gm6lj\" (UID: \"5480db01-47da-46d6-90d3-458cacdac16c\") " pod="kserve/s3-init-gm6lj"
Sep 17 18:09:49.598403 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:49.598327    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gm6lj"
Sep 17 18:09:49.866765 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:49.866737    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-gm6lj"]
Sep 17 18:09:49.868421 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:09:49.868390    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5480db01_47da_46d6_90d3_458cacdac16c.slice/crio-945d1605d17ddb5870628506ac2b8dd149e56a9f1d12f05c50f6ee3c72e84127 WatchSource:0}: Error finding container 945d1605d17ddb5870628506ac2b8dd149e56a9f1d12f05c50f6ee3c72e84127: Status 404 returned error can't find the container with id 945d1605d17ddb5870628506ac2b8dd149e56a9f1d12f05c50f6ee3c72e84127
Sep 17 18:09:50.519929 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:50.519887    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gm6lj" event={"ID":"5480db01-47da-46d6-90d3-458cacdac16c","Type":"ContainerStarted","Data":"945d1605d17ddb5870628506ac2b8dd149e56a9f1d12f05c50f6ee3c72e84127"}
Sep 17 18:09:54.546924 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:54.546881    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gm6lj" event={"ID":"5480db01-47da-46d6-90d3-458cacdac16c","Type":"ContainerStarted","Data":"727cbdc586de0d07ef13597932993b3edda43f46656c11b577bf5b8407eae4fd"}
Sep 17 18:09:55.553037 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:55.553010    2574 generic.go:356] "Generic (PLEG): container finished" podID="5480db01-47da-46d6-90d3-458cacdac16c" containerID="727cbdc586de0d07ef13597932993b3edda43f46656c11b577bf5b8407eae4fd" exitCode=0
Sep 17 18:09:55.553329 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:09:55.553091    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gm6lj" event={"ID":"5480db01-47da-46d6-90d3-458cacdac16c","Type":"ContainerDied","Data":"727cbdc586de0d07ef13597932993b3edda43f46656c11b577bf5b8407eae4fd"}
Sep 17 18:10:00.586610 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:10:00.586573    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gm6lj" event={"ID":"5480db01-47da-46d6-90d3-458cacdac16c","Type":"ContainerStarted","Data":"5bb4afa4941476410316b7466233a72de583e1b1a537d0bfafc764a1b276c1bb"}
Sep 17 18:10:00.604665 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:10:00.604612    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-gm6lj" podStartSLOduration=1.63940036 podStartE2EDuration="11.604596861s" totalImagesPullingTime="8.016211134s" totalInitContainerRuntime="1.948985367s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:00.602293142 +0000 UTC m=+770.903115482" watchObservedRunningTime="2026-09-17 18:10:00.604596861 +0000 UTC m=+770.905419185"
Sep 17 18:10:16.310619 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:10:16.310582    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:53.680779 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:10:53.680722    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:04.917757 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:04.917725    2574 generic.go:356] "Generic (PLEG): container finished" podID="5480db01-47da-46d6-90d3-458cacdac16c" containerID="5bb4afa4941476410316b7466233a72de583e1b1a537d0bfafc764a1b276c1bb" exitCode=0
Sep 17 18:11:04.918154 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:04.917797    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gm6lj" event={"ID":"5480db01-47da-46d6-90d3-458cacdac16c","Type":"ContainerDied","Data":"5bb4afa4941476410316b7466233a72de583e1b1a537d0bfafc764a1b276c1bb"}
Sep 17 18:11:06.051976 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:06.051951    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gm6lj"
Sep 17 18:11:06.187567 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:06.187477    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5480db01-47da-46d6-90d3-458cacdac16c-model-cache\") pod \"5480db01-47da-46d6-90d3-458cacdac16c\" (UID: \"5480db01-47da-46d6-90d3-458cacdac16c\") "
Sep 17 18:11:06.187703 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:06.187590    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-576xw\" (UniqueName: \"kubernetes.io/projected/5480db01-47da-46d6-90d3-458cacdac16c-kube-api-access-576xw\") pod \"5480db01-47da-46d6-90d3-458cacdac16c\" (UID: \"5480db01-47da-46d6-90d3-458cacdac16c\") "
Sep 17 18:11:06.189705 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:06.189683    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5480db01-47da-46d6-90d3-458cacdac16c-kube-api-access-576xw" (OuterVolumeSpecName: "kube-api-access-576xw") pod "5480db01-47da-46d6-90d3-458cacdac16c" (UID: "5480db01-47da-46d6-90d3-458cacdac16c"). InnerVolumeSpecName "kube-api-access-576xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:06.210457 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:06.210426    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5480db01-47da-46d6-90d3-458cacdac16c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5480db01-47da-46d6-90d3-458cacdac16c" (UID: "5480db01-47da-46d6-90d3-458cacdac16c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:06.288388 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:06.288350    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-576xw\" (UniqueName: \"kubernetes.io/projected/5480db01-47da-46d6-90d3-458cacdac16c-kube-api-access-576xw\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:11:06.288388 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:06.288382    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5480db01-47da-46d6-90d3-458cacdac16c-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:11:06.930124 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:06.930096    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gm6lj"
Sep 17 18:11:06.930124 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:06.930119    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gm6lj" event={"ID":"5480db01-47da-46d6-90d3-458cacdac16c","Type":"ContainerDied","Data":"945d1605d17ddb5870628506ac2b8dd149e56a9f1d12f05c50f6ee3c72e84127"}
Sep 17 18:11:06.930338 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:06.930154    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="945d1605d17ddb5870628506ac2b8dd149e56a9f1d12f05c50f6ee3c72e84127"
Sep 17 18:11:07.314562 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:07.314533    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:07.415048 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:07.415019    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:19.798108 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:19.798079    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:19.895293 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:19.895261    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:31.075834 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.075799    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"]
Sep 17 18:11:31.076241 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.076225    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5480db01-47da-46d6-90d3-458cacdac16c" containerName="s3-init"
Sep 17 18:11:31.076302 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.076243    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5480db01-47da-46d6-90d3-458cacdac16c" containerName="s3-init"
Sep 17 18:11:31.076302 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.076253    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5480db01-47da-46d6-90d3-458cacdac16c" containerName="download-hf-model"
Sep 17 18:11:31.076302 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.076259    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5480db01-47da-46d6-90d3-458cacdac16c" containerName="download-hf-model"
Sep 17 18:11:31.076440 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.076358    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5480db01-47da-46d6-90d3-458cacdac16c" containerName="s3-init"
Sep 17 18:11:31.078838 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.078820    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.082499 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.082479    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 18:11:31.082650 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.082485    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-2cjlh\""
Sep 17 18:11:31.082650 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.082566    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 18:11:31.082650 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.082642    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 18:11:31.088033 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.088010    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"]
Sep 17 18:11:31.109586 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.109543    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:31.188612 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.188581    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.188755 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.188619    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvcv\" (UniqueName: \"kubernetes.io/projected/322b208d-be6b-4700-a67e-38ba7f3a47c4-kube-api-access-mmvcv\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.188755 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.188637    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.188755 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.188687    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322b208d-be6b-4700-a67e-38ba7f3a47c4-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.188755 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.188718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.188898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.188774    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.289680 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.289640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvcv\" (UniqueName: \"kubernetes.io/projected/322b208d-be6b-4700-a67e-38ba7f3a47c4-kube-api-access-mmvcv\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.289680 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.289682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.289896 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.289710    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322b208d-be6b-4700-a67e-38ba7f3a47c4-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.289896 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.289738    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.289896 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.289807    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.289896 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.289887    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.290164 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.290139    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.290243 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.290145    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.290243 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.290213    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.292152 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.292131    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.292409 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.292391    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322b208d-be6b-4700-a67e-38ba7f3a47c4-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.298101 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.298078    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvcv\" (UniqueName: \"kubernetes.io/projected/322b208d-be6b-4700-a67e-38ba7f3a47c4-kube-api-access-mmvcv\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.390402 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.390336    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:31.498408 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.498378    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"]
Sep 17 18:11:31.500941 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.500923    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"
Sep 17 18:11:31.503363 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.503341    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-thp6p\""
Sep 17 18:11:31.515108 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.515078    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"]
Sep 17 18:11:31.524019 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.523995    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:31.592107 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.592084    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed97b53-fefe-47f5-941b-3f4984885fdd-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl\" (UID: \"4ed97b53-fefe-47f5-941b-3f4984885fdd\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"
Sep 17 18:11:31.592213 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.592114    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9gr6\" (UniqueName: \"kubernetes.io/projected/4ed97b53-fefe-47f5-941b-3f4984885fdd-kube-api-access-h9gr6\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl\" (UID: \"4ed97b53-fefe-47f5-941b-3f4984885fdd\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"
Sep 17 18:11:31.660787 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.660767    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"]
Sep 17 18:11:31.662215 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:11:31.662194    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322b208d_be6b_4700_a67e_38ba7f3a47c4.slice/crio-053503ab3ab659ec63d834903c1abc2dcfded490f07fb74cace468f4c4aa3774 WatchSource:0}: Error finding container 053503ab3ab659ec63d834903c1abc2dcfded490f07fb74cace468f4c4aa3774: Status 404 returned error can't find the container with id 053503ab3ab659ec63d834903c1abc2dcfded490f07fb74cace468f4c4aa3774
Sep 17 18:11:31.692773 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.692744    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed97b53-fefe-47f5-941b-3f4984885fdd-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl\" (UID: \"4ed97b53-fefe-47f5-941b-3f4984885fdd\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"
Sep 17 18:11:31.692874 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.692779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9gr6\" (UniqueName: \"kubernetes.io/projected/4ed97b53-fefe-47f5-941b-3f4984885fdd-kube-api-access-h9gr6\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl\" (UID: \"4ed97b53-fefe-47f5-941b-3f4984885fdd\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"
Sep 17 18:11:31.695342 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.695319    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed97b53-fefe-47f5-941b-3f4984885fdd-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl\" (UID: \"4ed97b53-fefe-47f5-941b-3f4984885fdd\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"
Sep 17 18:11:31.700737 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.700713    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9gr6\" (UniqueName: \"kubernetes.io/projected/4ed97b53-fefe-47f5-941b-3f4984885fdd-kube-api-access-h9gr6\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl\" (UID: \"4ed97b53-fefe-47f5-941b-3f4984885fdd\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"
Sep 17 18:11:31.811597 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:31.811561    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"
Sep 17 18:11:32.065785 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:32.065753    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk" event={"ID":"322b208d-be6b-4700-a67e-38ba7f3a47c4","Type":"ContainerStarted","Data":"053503ab3ab659ec63d834903c1abc2dcfded490f07fb74cace468f4c4aa3774"}
Sep 17 18:11:32.089146 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:32.089072    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"]
Sep 17 18:11:32.118700 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:11:32.118667    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed97b53_fefe_47f5_941b_3f4984885fdd.slice/crio-45bd8f8cfdda2a7c786377933ce9f0b129e88b9fb55fb941043e238addb1a5bf WatchSource:0}: Error finding container 45bd8f8cfdda2a7c786377933ce9f0b129e88b9fb55fb941043e238addb1a5bf: Status 404 returned error can't find the container with id 45bd8f8cfdda2a7c786377933ce9f0b129e88b9fb55fb941043e238addb1a5bf
Sep 17 18:11:33.075252 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:33.075201    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl" event={"ID":"4ed97b53-fefe-47f5-941b-3f4984885fdd","Type":"ContainerStarted","Data":"45bd8f8cfdda2a7c786377933ce9f0b129e88b9fb55fb941043e238addb1a5bf"}
Sep 17 18:11:34.083157 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:34.083105    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk" event={"ID":"322b208d-be6b-4700-a67e-38ba7f3a47c4","Type":"ContainerStarted","Data":"f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037"}
Sep 17 18:11:34.103530 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:34.103448    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk" podStartSLOduration=1.548551163 podStartE2EDuration="3.103425173s" totalImagesPullingTime="1.55487401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:34.100424122 +0000 UTC m=+864.401246448" watchObservedRunningTime="2026-09-17 18:11:34.103425173 +0000 UTC m=+864.404247500"
Sep 17 18:11:34.391466 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:34.391373    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:35.096445 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:35.096406    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl" event={"ID":"4ed97b53-fefe-47f5-941b-3f4984885fdd","Type":"ContainerStarted","Data":"4abc9c912a868eddb0ebf471cc537325bb80ed5d4af39939f4dd62da238a8f30"}
Sep 17 18:11:35.096892 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:35.096687    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"
Sep 17 18:11:35.116217 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:35.116158    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl" podStartSLOduration=1.610748115 podStartE2EDuration="4.11614188s" totalImagesPullingTime="2.505393765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:35.114890429 +0000 UTC m=+865.415712763" watchObservedRunningTime="2026-09-17 18:11:35.11614188 +0000 UTC m=+865.416964203"
Sep 17 18:11:41.391464 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:41.391421    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:41.404618 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:41.404589    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:42.147300 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:42.147275    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:11:57.973362 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:11:57.973285    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:06.104012 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:06.103972    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"
Sep 17 18:12:06.987042 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:06.987002    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:07.199335 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:07.199302    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:11.816059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:11.816029    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:12:11.818215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:11.818193    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:12:17.611211 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:17.611167    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:19.382556 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.382503    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"]
Sep 17 18:12:19.387548 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.387489    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"]
Sep 17 18:12:19.392579 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:12:19.392537    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:12:19.392579 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:12:19.392579 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:12:19.392579 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037" cmd=["/bin/sleep","15"]
Sep 17 18:12:19.392835 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.392611    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk" message="" err=<
Sep 17 18:12:19.392835 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:12:19.392835 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:12:19.392835 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 18:12:19.392835 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.392615    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:19.392835 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:12:19.392630    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:12:19.392835 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:12:19.392835 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:12:19.392835 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk" podUID="322b208d-be6b-4700-a67e-38ba7f3a47c4" containerName="main" containerID="cri-o://f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037"
Sep 17 18:12:19.392835 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.392678    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk" podUID="322b208d-be6b-4700-a67e-38ba7f3a47c4" containerName="main" containerID="cri-o://f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037" gracePeriod=60
Sep 17 18:12:19.484308 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.484276    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:19.748390 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.748362    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:12:19.822311 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.822277    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-tmp-dir\") pod \"322b208d-be6b-4700-a67e-38ba7f3a47c4\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") "
Sep 17 18:12:19.822311 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.822314    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-model-cache\") pod \"322b208d-be6b-4700-a67e-38ba7f3a47c4\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") "
Sep 17 18:12:19.822567 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.822336    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-dshm\") pod \"322b208d-be6b-4700-a67e-38ba7f3a47c4\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") "
Sep 17 18:12:19.822567 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.822370    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322b208d-be6b-4700-a67e-38ba7f3a47c4-tls-certs\") pod \"322b208d-be6b-4700-a67e-38ba7f3a47c4\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") "
Sep 17 18:12:19.822567 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.822452    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmvcv\" (UniqueName: \"kubernetes.io/projected/322b208d-be6b-4700-a67e-38ba7f3a47c4-kube-api-access-mmvcv\") pod \"322b208d-be6b-4700-a67e-38ba7f3a47c4\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") "
Sep 17 18:12:19.822567 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.822501    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-home\") pod \"322b208d-be6b-4700-a67e-38ba7f3a47c4\" (UID: \"322b208d-be6b-4700-a67e-38ba7f3a47c4\") "
Sep 17 18:12:19.822806 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.822563    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "322b208d-be6b-4700-a67e-38ba7f3a47c4" (UID: "322b208d-be6b-4700-a67e-38ba7f3a47c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:19.822806 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.822618    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "322b208d-be6b-4700-a67e-38ba7f3a47c4" (UID: "322b208d-be6b-4700-a67e-38ba7f3a47c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:19.822806 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.822791    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-home" (OuterVolumeSpecName: "home") pod "322b208d-be6b-4700-a67e-38ba7f3a47c4" (UID: "322b208d-be6b-4700-a67e-38ba7f3a47c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:19.822806 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.822799    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:12:19.822978 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.822823    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:12:19.825029 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.825005    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322b208d-be6b-4700-a67e-38ba7f3a47c4-kube-api-access-mmvcv" (OuterVolumeSpecName: "kube-api-access-mmvcv") pod "322b208d-be6b-4700-a67e-38ba7f3a47c4" (UID: "322b208d-be6b-4700-a67e-38ba7f3a47c4"). InnerVolumeSpecName "kube-api-access-mmvcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:19.825029 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.825015    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322b208d-be6b-4700-a67e-38ba7f3a47c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "322b208d-be6b-4700-a67e-38ba7f3a47c4" (UID: "322b208d-be6b-4700-a67e-38ba7f3a47c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:19.825171 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.825006    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-dshm" (OuterVolumeSpecName: "dshm") pod "322b208d-be6b-4700-a67e-38ba7f3a47c4" (UID: "322b208d-be6b-4700-a67e-38ba7f3a47c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:19.923592 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.923476    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:12:19.923592 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.923504    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/322b208d-be6b-4700-a67e-38ba7f3a47c4-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:12:19.923592 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.923538    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/322b208d-be6b-4700-a67e-38ba7f3a47c4-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:12:19.923592 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:19.923546    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmvcv\" (UniqueName: \"kubernetes.io/projected/322b208d-be6b-4700-a67e-38ba7f3a47c4-kube-api-access-mmvcv\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:12:20.347265 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:20.347229    2574 generic.go:356] "Generic (PLEG): container finished" podID="322b208d-be6b-4700-a67e-38ba7f3a47c4" containerID="f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037" exitCode=0
Sep 17 18:12:20.347440 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:20.347276    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"
Sep 17 18:12:20.347440 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:20.347284    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk" event={"ID":"322b208d-be6b-4700-a67e-38ba7f3a47c4","Type":"ContainerDied","Data":"f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037"}
Sep 17 18:12:20.347440 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:20.347329    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk" event={"ID":"322b208d-be6b-4700-a67e-38ba7f3a47c4","Type":"ContainerDied","Data":"053503ab3ab659ec63d834903c1abc2dcfded490f07fb74cace468f4c4aa3774"}
Sep 17 18:12:20.347440 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:20.347351    2574 scope.go:117] "RemoveContainer" containerID="f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037"
Sep 17 18:12:20.356747 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:20.356725    2574 scope.go:117] "RemoveContainer" containerID="f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037"
Sep 17 18:12:20.357025 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:12:20.357007    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037\": container with ID starting with f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037 not found: ID does not exist" containerID="f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037"
Sep 17 18:12:20.357076 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:20.357034    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037"} err="failed to get container status \"f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037\": rpc error: code = NotFound desc = could not find container \"f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037\": container with ID starting with f07adb636fafe17b497f3750f6f882e6cf663be2411b4cefb35d2db372039037 not found: ID does not exist"
Sep 17 18:12:20.363772 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:20.363749    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"]
Sep 17 18:12:20.367126 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:20.367098    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-lp4bk"]
Sep 17 18:12:21.939095 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:21.939040    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="322b208d-be6b-4700-a67e-38ba7f3a47c4" path="/var/lib/kubelet/pods/322b208d-be6b-4700-a67e-38ba7f3a47c4/volumes"
Sep 17 18:12:24.057124 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:24.057090    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:26.104563 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:26.104498    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl" podUID="4ed97b53-fefe-47f5-941b-3f4984885fdd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:30.944906 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:30.944567    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 17 18:12:30.944906 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:12:30.944612    2574 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl" podUID="4ed97b53-fefe-47f5-941b-3f4984885fdd" containerName="main" containerID="cri-o://4abc9c912a868eddb0ebf471cc537325bb80ed5d4af39939f4dd62da238a8f30"
Sep 17 18:12:30.944906 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:30.944681    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl" podUID="4ed97b53-fefe-47f5-941b-3f4984885fdd" containerName="main" containerID="cri-o://4abc9c912a868eddb0ebf471cc537325bb80ed5d4af39939f4dd62da238a8f30" gracePeriod=49
Sep 17 18:12:31.096173 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.096146    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl_4ed97b53-fefe-47f5-941b-3f4984885fdd/main/0.log"
Sep 17 18:12:31.098522 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.098492    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"
Sep 17 18:12:31.205650 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.205547    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9gr6\" (UniqueName: \"kubernetes.io/projected/4ed97b53-fefe-47f5-941b-3f4984885fdd-kube-api-access-h9gr6\") pod \"4ed97b53-fefe-47f5-941b-3f4984885fdd\" (UID: \"4ed97b53-fefe-47f5-941b-3f4984885fdd\") "
Sep 17 18:12:31.205650 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.205628    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed97b53-fefe-47f5-941b-3f4984885fdd-tls-certs\") pod \"4ed97b53-fefe-47f5-941b-3f4984885fdd\" (UID: \"4ed97b53-fefe-47f5-941b-3f4984885fdd\") "
Sep 17 18:12:31.207923 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.207886    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed97b53-fefe-47f5-941b-3f4984885fdd-kube-api-access-h9gr6" (OuterVolumeSpecName: "kube-api-access-h9gr6") pod "4ed97b53-fefe-47f5-941b-3f4984885fdd" (UID: "4ed97b53-fefe-47f5-941b-3f4984885fdd"). InnerVolumeSpecName "kube-api-access-h9gr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:31.208030 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.207931    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ed97b53-fefe-47f5-941b-3f4984885fdd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ed97b53-fefe-47f5-941b-3f4984885fdd" (UID: "4ed97b53-fefe-47f5-941b-3f4984885fdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:31.307195 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.307158    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9gr6\" (UniqueName: \"kubernetes.io/projected/4ed97b53-fefe-47f5-941b-3f4984885fdd-kube-api-access-h9gr6\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:12:31.307195 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.307189    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ed97b53-fefe-47f5-941b-3f4984885fdd-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:12:31.414045 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.414015    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl_4ed97b53-fefe-47f5-941b-3f4984885fdd/main/0.log"
Sep 17 18:12:31.414222 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.414062    2574 generic.go:356] "Generic (PLEG): container finished" podID="4ed97b53-fefe-47f5-941b-3f4984885fdd" containerID="4abc9c912a868eddb0ebf471cc537325bb80ed5d4af39939f4dd62da238a8f30" exitCode=1
Sep 17 18:12:31.414222 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.414099    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl" event={"ID":"4ed97b53-fefe-47f5-941b-3f4984885fdd","Type":"ContainerDied","Data":"4abc9c912a868eddb0ebf471cc537325bb80ed5d4af39939f4dd62da238a8f30"}
Sep 17 18:12:31.414222 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.414124    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"
Sep 17 18:12:31.414222 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.414158    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl" event={"ID":"4ed97b53-fefe-47f5-941b-3f4984885fdd","Type":"ContainerDied","Data":"45bd8f8cfdda2a7c786377933ce9f0b129e88b9fb55fb941043e238addb1a5bf"}
Sep 17 18:12:31.414222 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.414184    2574 scope.go:117] "RemoveContainer" containerID="4abc9c912a868eddb0ebf471cc537325bb80ed5d4af39939f4dd62da238a8f30"
Sep 17 18:12:31.423770 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.423752    2574 scope.go:117] "RemoveContainer" containerID="4abc9c912a868eddb0ebf471cc537325bb80ed5d4af39939f4dd62da238a8f30"
Sep 17 18:12:31.424042 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:12:31.424024    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abc9c912a868eddb0ebf471cc537325bb80ed5d4af39939f4dd62da238a8f30\": container with ID starting with 4abc9c912a868eddb0ebf471cc537325bb80ed5d4af39939f4dd62da238a8f30 not found: ID does not exist" containerID="4abc9c912a868eddb0ebf471cc537325bb80ed5d4af39939f4dd62da238a8f30"
Sep 17 18:12:31.424088 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.424050    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abc9c912a868eddb0ebf471cc537325bb80ed5d4af39939f4dd62da238a8f30"} err="failed to get container status \"4abc9c912a868eddb0ebf471cc537325bb80ed5d4af39939f4dd62da238a8f30\": rpc error: code = NotFound desc = could not find container \"4abc9c912a868eddb0ebf471cc537325bb80ed5d4af39939f4dd62da238a8f30\": container with ID starting with 4abc9c912a868eddb0ebf471cc537325bb80ed5d4af39939f4dd62da238a8f30 not found: ID does not exist"
Sep 17 18:12:31.435292 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.435266    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"]
Sep 17 18:12:31.441061 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.441039    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc526gsl"]
Sep 17 18:12:31.937860 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:31.937824    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ed97b53-fefe-47f5-941b-3f4984885fdd" path="/var/lib/kubelet/pods/4ed97b53-fefe-47f5-941b-3f4984885fdd/volumes"
Sep 17 18:12:35.426933 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:35.426898    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:35.528408 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:35.528371    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:45.704164 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:45.704120    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:45.778562 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:12:45.778504    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:10.700381 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:10.700341    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:10.794428 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:10.794395    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:16.976035 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:16.976001    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:22.671655 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.671571    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"]
Sep 17 18:13:22.672150 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.672134    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322b208d-be6b-4700-a67e-38ba7f3a47c4" containerName="main"
Sep 17 18:13:22.672193 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.672154    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="322b208d-be6b-4700-a67e-38ba7f3a47c4" containerName="main"
Sep 17 18:13:22.672193 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.672187    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ed97b53-fefe-47f5-941b-3f4984885fdd" containerName="main"
Sep 17 18:13:22.672263 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.672195    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ed97b53-fefe-47f5-941b-3f4984885fdd" containerName="main"
Sep 17 18:13:22.672300 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.672280    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ed97b53-fefe-47f5-941b-3f4984885fdd" containerName="main"
Sep 17 18:13:22.672300 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.672297    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="322b208d-be6b-4700-a67e-38ba7f3a47c4" containerName="main"
Sep 17 18:13:22.677193 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.677171    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.679928 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.679893    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 18:13:22.680817 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.680794    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-mrfdh\""
Sep 17 18:13:22.680932 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.680796    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 18:13:22.680932 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.680855    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 18:13:22.687781 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.687754    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"]
Sep 17 18:13:22.722830 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.722788    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:22.847434 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.847387    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.847434 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.847438    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.847694 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.847463    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.847694 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.847559    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.847694 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.847587    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r65rv\" (UniqueName: \"kubernetes.io/projected/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-kube-api-access-r65rv\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.847694 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.847604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.949053 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.948961    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.949053 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.949010    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r65rv\" (UniqueName: \"kubernetes.io/projected/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-kube-api-access-r65rv\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.949053 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.949029    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.949053 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.949056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.949440 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.949075    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.949440 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.949097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.949571 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.949549    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.949571 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.949561    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.949683 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.949619    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.951581 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.951562    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.951687 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.951673    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.959860 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.959839    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r65rv\" (UniqueName: \"kubernetes.io/projected/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-kube-api-access-r65rv\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-dms6v\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.989798 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.989764    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:22.999374 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:22.999348    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:23.263435 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:23.263406    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"]
Sep 17 18:13:23.265532 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:13:23.265491    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6d2b7d_76db_4a66_81ca_b995c5ecfc48.slice/crio-547e87c9ea69c7efe5a1f47aa457ecf396c89957722316bedbb815fb9d20e1cc WatchSource:0}: Error finding container 547e87c9ea69c7efe5a1f47aa457ecf396c89957722316bedbb815fb9d20e1cc: Status 404 returned error can't find the container with id 547e87c9ea69c7efe5a1f47aa457ecf396c89957722316bedbb815fb9d20e1cc
Sep 17 18:13:23.715444 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:23.715336    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v" event={"ID":"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48","Type":"ContainerStarted","Data":"3e01eec58e13bad40b41a2f671c5de3421a280844c5861128e2c03c486c664c0"}
Sep 17 18:13:23.715444 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:23.715410    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v" event={"ID":"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48","Type":"ContainerStarted","Data":"547e87c9ea69c7efe5a1f47aa457ecf396c89957722316bedbb815fb9d20e1cc"}
Sep 17 18:13:23.736615 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:23.736556    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v" podStartSLOduration=1.7365406 podStartE2EDuration="1.7365406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:23.735931179 +0000 UTC m=+974.036753503" watchObservedRunningTime="2026-09-17 18:13:23.7365406 +0000 UTC m=+974.037362922"
Sep 17 18:13:23.989927 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:23.989838    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:24.044302 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:24.044266    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:24.142758 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:24.142721    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:28.361010 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:28.360970    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:32.990368 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:32.990329    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:33.003316 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:33.003288    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:33.777871 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:33.777839    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:13:35.951818 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:35.951778    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:36.045925 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:36.045885    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:39.898427 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:39.898391    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:40.177262 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.177173    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"]
Sep 17 18:13:40.180826 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.180800    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"
Sep 17 18:13:40.184275 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.184245    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 18:13:40.184404 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.184290    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 18:13:40.184404 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.184245    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-8l29w\""
Sep 17 18:13:40.184404 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.184261    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-fs2js\""
Sep 17 18:13:40.184404 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.184364    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 18:13:40.187865 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.187842    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"]
Sep 17 18:13:40.202154 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.202125    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:40.293165 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.293135    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07de62a1-95ce-4844-ae9b-522fb83963f8-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5\" (UID: \"07de62a1-95ce-4844-ae9b-522fb83963f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"
Sep 17 18:13:40.293345 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.293177    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr5vp\" (UniqueName: \"kubernetes.io/projected/07de62a1-95ce-4844-ae9b-522fb83963f8-kube-api-access-zr5vp\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5\" (UID: \"07de62a1-95ce-4844-ae9b-522fb83963f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"
Sep 17 18:13:40.394380 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.394341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07de62a1-95ce-4844-ae9b-522fb83963f8-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5\" (UID: \"07de62a1-95ce-4844-ae9b-522fb83963f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"
Sep 17 18:13:40.394554 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.394391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr5vp\" (UniqueName: \"kubernetes.io/projected/07de62a1-95ce-4844-ae9b-522fb83963f8-kube-api-access-zr5vp\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5\" (UID: \"07de62a1-95ce-4844-ae9b-522fb83963f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"
Sep 17 18:13:40.397124 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.397101    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07de62a1-95ce-4844-ae9b-522fb83963f8-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5\" (UID: \"07de62a1-95ce-4844-ae9b-522fb83963f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"
Sep 17 18:13:40.404141 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.404114    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr5vp\" (UniqueName: \"kubernetes.io/projected/07de62a1-95ce-4844-ae9b-522fb83963f8-kube-api-access-zr5vp\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5\" (UID: \"07de62a1-95ce-4844-ae9b-522fb83963f8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"
Sep 17 18:13:40.493160 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.493071    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"
Sep 17 18:13:40.769587 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.769555    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"]
Sep 17 18:13:40.771131 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:13:40.771105    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07de62a1_95ce_4844_ae9b_522fb83963f8.slice/crio-08bcbcbd4cd8b2436318df961ecffca49df1b42b72626d822241f395d00a3fcf WatchSource:0}: Error finding container 08bcbcbd4cd8b2436318df961ecffca49df1b42b72626d822241f395d00a3fcf: Status 404 returned error can't find the container with id 08bcbcbd4cd8b2436318df961ecffca49df1b42b72626d822241f395d00a3fcf
Sep 17 18:13:40.806120 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:40.805999    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5" event={"ID":"07de62a1-95ce-4844-ae9b-522fb83963f8","Type":"ContainerStarted","Data":"08bcbcbd4cd8b2436318df961ecffca49df1b42b72626d822241f395d00a3fcf"}
Sep 17 18:13:41.813784 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:41.813747    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5" event={"ID":"07de62a1-95ce-4844-ae9b-522fb83963f8","Type":"ContainerStarted","Data":"41f2997cec7dd3f7320d7598610be2f307fc4dbfc45f4f3f3634ec3f48958ca3"}
Sep 17 18:13:41.814209 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:41.813903    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"
Sep 17 18:13:41.830732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:41.830679    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5" podStartSLOduration=1.830660896 podStartE2EDuration="1.830660896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:41.830368157 +0000 UTC m=+992.131190481" watchObservedRunningTime="2026-09-17 18:13:41.830660896 +0000 UTC m=+992.131483223"
Sep 17 18:13:42.913288 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:42.913228    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:48.685181 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:48.685100    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:48.786222 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:48.786179    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:55.012323 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:55.012281    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:59.628774 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:59.628740    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"]
Sep 17 18:13:59.632356 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:59.632319    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:59.638847 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:13:59.638809    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:13:59.638847 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:13:59.638847 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:13:59.638847 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="3e01eec58e13bad40b41a2f671c5de3421a280844c5861128e2c03c486c664c0" cmd=["/bin/sleep","15"]
Sep 17 18:13:59.639099 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:59.638873    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v" message="" err=<
Sep 17 18:13:59.639099 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:13:59.639099 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:13:59.639099 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 18:13:59.639099 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:13:59.638890    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:13:59.639099 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:13:59.639099 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:13:59.639099 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v" podUID="0e6d2b7d-76db-4a66-81ca-b995c5ecfc48" containerName="main" containerID="cri-o://3e01eec58e13bad40b41a2f671c5de3421a280844c5861128e2c03c486c664c0"
Sep 17 18:13:59.639099 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:59.638949    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v" podUID="0e6d2b7d-76db-4a66-81ca-b995c5ecfc48" containerName="main" containerID="cri-o://3e01eec58e13bad40b41a2f671c5de3421a280844c5861128e2c03c486c664c0" gracePeriod=60
Sep 17 18:13:59.694459 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:59.694421    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:59.766630 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:59.766590    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v" podUID="0e6d2b7d-76db-4a66-81ca-b995c5ecfc48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:13:59.934124 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:59.934039    2574 generic.go:356] "Generic (PLEG): container finished" podID="0e6d2b7d-76db-4a66-81ca-b995c5ecfc48" containerID="3e01eec58e13bad40b41a2f671c5de3421a280844c5861128e2c03c486c664c0" exitCode=0
Sep 17 18:13:59.938035 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:13:59.938003    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v" event={"ID":"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48","Type":"ContainerDied","Data":"3e01eec58e13bad40b41a2f671c5de3421a280844c5861128e2c03c486c664c0"}
Sep 17 18:14:00.395031 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.395005    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:14:00.477883 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.477850    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-home\") pod \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") "
Sep 17 18:14:00.478081 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.477907    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-dshm\") pod \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") "
Sep 17 18:14:00.478081 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.477957    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-tls-certs\") pod \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") "
Sep 17 18:14:00.478081 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.477985    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r65rv\" (UniqueName: \"kubernetes.io/projected/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-kube-api-access-r65rv\") pod \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") "
Sep 17 18:14:00.478081 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.478003    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-tmp-dir\") pod \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") "
Sep 17 18:14:00.478081 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.478039    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-model-cache\") pod \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\" (UID: \"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48\") "
Sep 17 18:14:00.478408 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.478152    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-home" (OuterVolumeSpecName: "home") pod "0e6d2b7d-76db-4a66-81ca-b995c5ecfc48" (UID: "0e6d2b7d-76db-4a66-81ca-b995c5ecfc48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:00.478408 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.478298    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:14:00.478408 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.478349    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e6d2b7d-76db-4a66-81ca-b995c5ecfc48" (UID: "0e6d2b7d-76db-4a66-81ca-b995c5ecfc48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:00.478408 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.478358    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e6d2b7d-76db-4a66-81ca-b995c5ecfc48" (UID: "0e6d2b7d-76db-4a66-81ca-b995c5ecfc48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:00.480425 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.480393    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-dshm" (OuterVolumeSpecName: "dshm") pod "0e6d2b7d-76db-4a66-81ca-b995c5ecfc48" (UID: "0e6d2b7d-76db-4a66-81ca-b995c5ecfc48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:00.480837 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.480809    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e6d2b7d-76db-4a66-81ca-b995c5ecfc48" (UID: "0e6d2b7d-76db-4a66-81ca-b995c5ecfc48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:00.480932 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.480831    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-kube-api-access-r65rv" (OuterVolumeSpecName: "kube-api-access-r65rv") pod "0e6d2b7d-76db-4a66-81ca-b995c5ecfc48" (UID: "0e6d2b7d-76db-4a66-81ca-b995c5ecfc48"). InnerVolumeSpecName "kube-api-access-r65rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:00.578979 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.578876    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:14:00.578979 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.578931    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:14:00.578979 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.578943    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r65rv\" (UniqueName: \"kubernetes.io/projected/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-kube-api-access-r65rv\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:14:00.578979 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.578953    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:14:00.578979 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.578963    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:14:00.941267 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.941181    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"
Sep 17 18:14:00.941267 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.941194    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v" event={"ID":"0e6d2b7d-76db-4a66-81ca-b995c5ecfc48","Type":"ContainerDied","Data":"547e87c9ea69c7efe5a1f47aa457ecf396c89957722316bedbb815fb9d20e1cc"}
Sep 17 18:14:00.941771 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.941280    2574 scope.go:117] "RemoveContainer" containerID="3e01eec58e13bad40b41a2f671c5de3421a280844c5861128e2c03c486c664c0"
Sep 17 18:14:00.962764 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.962725    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"]
Sep 17 18:14:00.967004 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:00.966980    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-dms6v"]
Sep 17 18:14:01.939141 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:01.939104    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e6d2b7d-76db-4a66-81ca-b995c5ecfc48" path="/var/lib/kubelet/pods/0e6d2b7d-76db-4a66-81ca-b995c5ecfc48/volumes"
Sep 17 18:14:12.820758 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:12.820723    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"
Sep 17 18:14:19.892425 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:19.892357    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:19.905914 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:19.905875    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"]
Sep 17 18:14:19.996466 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:19.996427    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:22.821091 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:22.821049    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5" podUID="07de62a1-95ce-4844-ae9b-522fb83963f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:32.820138 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:32.820096    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5" podUID="07de62a1-95ce-4844-ae9b-522fb83963f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:34.926346 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:34.926306    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5" podUID="07de62a1-95ce-4844-ae9b-522fb83963f8" containerName="main" containerID="cri-o://41f2997cec7dd3f7320d7598610be2f307fc4dbfc45f4f3f3634ec3f48958ca3" gracePeriod=45
Sep 17 18:14:42.820555 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:42.820487    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5" podUID="07de62a1-95ce-4844-ae9b-522fb83963f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:42.820947 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:42.820634    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"
Sep 17 18:14:47.728972 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:47.728938    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:48.757811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:48.757777    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:52.820868 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:52.820788    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5" podUID="07de62a1-95ce-4844-ae9b-522fb83963f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:52.901074 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:52.901033    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"]
Sep 17 18:14:52.901543 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:52.901493    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e6d2b7d-76db-4a66-81ca-b995c5ecfc48" containerName="main"
Sep 17 18:14:52.901543 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:52.901534    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e6d2b7d-76db-4a66-81ca-b995c5ecfc48" containerName="main"
Sep 17 18:14:52.901760 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:52.901619    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e6d2b7d-76db-4a66-81ca-b995c5ecfc48" containerName="main"
Sep 17 18:14:52.904788 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:52.904764    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:52.908171 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:52.908064    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 18:14:52.908171 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:52.908070    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 18:14:52.908171 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:52.908123    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 18:14:52.908171 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:52.908166    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-2nmgt\""
Sep 17 18:14:52.915840 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:52.915809    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"]
Sep 17 18:14:52.944676 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:52.944646    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:53.032286 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.032248    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.032467 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.032293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6sb8\" (UniqueName: \"kubernetes.io/projected/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-kube-api-access-g6sb8\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.032467 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.032369    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.032467 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.032417    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.032467 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.032447    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.032638 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.032548    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.133111 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.133026    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.133111 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.133080    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.133343 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.133112    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.133343 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.133140    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.133343 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.133189    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.133343 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.133220    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6sb8\" (UniqueName: \"kubernetes.io/projected/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-kube-api-access-g6sb8\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.133586 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.133443    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.133586 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.133485    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.133662 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.133613    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.135704 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.135683    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.135877 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.135860    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.140223 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.140204    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6sb8\" (UniqueName: \"kubernetes.io/projected/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-kube-api-access-g6sb8\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.145093 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.145065    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:53.217470 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.217436    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:53.493468 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:53.493443    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"]
Sep 17 18:14:53.495494 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:14:53.495466    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dab5c28_7745_4e3c_a45e_a7a831d44ef6.slice/crio-27e987e2691be90906b8f4d2c1e1eeee4a7bd02c3b4cd0c92a88f01f87edc65f WatchSource:0}: Error finding container 27e987e2691be90906b8f4d2c1e1eeee4a7bd02c3b4cd0c92a88f01f87edc65f: Status 404 returned error can't find the container with id 27e987e2691be90906b8f4d2c1e1eeee4a7bd02c3b4cd0c92a88f01f87edc65f
Sep 17 18:14:54.226883 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:54.226836    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p" event={"ID":"8dab5c28-7745-4e3c-a45e-a7a831d44ef6","Type":"ContainerStarted","Data":"1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2"}
Sep 17 18:14:54.227275 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:54.226897    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p" event={"ID":"8dab5c28-7745-4e3c-a45e-a7a831d44ef6","Type":"ContainerStarted","Data":"27e987e2691be90906b8f4d2c1e1eeee4a7bd02c3b4cd0c92a88f01f87edc65f"}
Sep 17 18:14:54.244883 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:54.244835    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p" podStartSLOduration=2.2448202 podStartE2EDuration="2.2448202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:54.243703143 +0000 UTC m=+1064.544525468" watchObservedRunningTime="2026-09-17 18:14:54.2448202 +0000 UTC m=+1064.545642541"
Sep 17 18:14:55.218060 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:55.218024    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:14:55.636449 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:14:55.636407    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:01.440189 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:01.440135    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:01.541592 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:01.541496    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:02.820966 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:02.820925    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5" podUID="07de62a1-95ce-4844-ae9b-522fb83963f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:03.218175 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:03.218081    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:15:03.231133 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:03.231103    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:15:03.286223 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:03.286184    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:15:05.181622 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.181594    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"
Sep 17 18:15:05.286718 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.286686    2574 generic.go:356] "Generic (PLEG): container finished" podID="07de62a1-95ce-4844-ae9b-522fb83963f8" containerID="41f2997cec7dd3f7320d7598610be2f307fc4dbfc45f4f3f3634ec3f48958ca3" exitCode=0
Sep 17 18:15:05.286910 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.286733    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"
Sep 17 18:15:05.286910 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.286772    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5" event={"ID":"07de62a1-95ce-4844-ae9b-522fb83963f8","Type":"ContainerDied","Data":"41f2997cec7dd3f7320d7598610be2f307fc4dbfc45f4f3f3634ec3f48958ca3"}
Sep 17 18:15:05.286910 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.286833    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5" event={"ID":"07de62a1-95ce-4844-ae9b-522fb83963f8","Type":"ContainerDied","Data":"08bcbcbd4cd8b2436318df961ecffca49df1b42b72626d822241f395d00a3fcf"}
Sep 17 18:15:05.286910 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.286864    2574 scope.go:117] "RemoveContainer" containerID="41f2997cec7dd3f7320d7598610be2f307fc4dbfc45f4f3f3634ec3f48958ca3"
Sep 17 18:15:05.295729 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.295713    2574 scope.go:117] "RemoveContainer" containerID="41f2997cec7dd3f7320d7598610be2f307fc4dbfc45f4f3f3634ec3f48958ca3"
Sep 17 18:15:05.296023 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:15:05.296002    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f2997cec7dd3f7320d7598610be2f307fc4dbfc45f4f3f3634ec3f48958ca3\": container with ID starting with 41f2997cec7dd3f7320d7598610be2f307fc4dbfc45f4f3f3634ec3f48958ca3 not found: ID does not exist" containerID="41f2997cec7dd3f7320d7598610be2f307fc4dbfc45f4f3f3634ec3f48958ca3"
Sep 17 18:15:05.296091 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.296035    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f2997cec7dd3f7320d7598610be2f307fc4dbfc45f4f3f3634ec3f48958ca3"} err="failed to get container status \"41f2997cec7dd3f7320d7598610be2f307fc4dbfc45f4f3f3634ec3f48958ca3\": rpc error: code = NotFound desc = could not find container \"41f2997cec7dd3f7320d7598610be2f307fc4dbfc45f4f3f3634ec3f48958ca3\": container with ID starting with 41f2997cec7dd3f7320d7598610be2f307fc4dbfc45f4f3f3634ec3f48958ca3 not found: ID does not exist"
Sep 17 18:15:05.343471 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.343446    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr5vp\" (UniqueName: \"kubernetes.io/projected/07de62a1-95ce-4844-ae9b-522fb83963f8-kube-api-access-zr5vp\") pod \"07de62a1-95ce-4844-ae9b-522fb83963f8\" (UID: \"07de62a1-95ce-4844-ae9b-522fb83963f8\") "
Sep 17 18:15:05.343664 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.343493    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07de62a1-95ce-4844-ae9b-522fb83963f8-tls-certs\") pod \"07de62a1-95ce-4844-ae9b-522fb83963f8\" (UID: \"07de62a1-95ce-4844-ae9b-522fb83963f8\") "
Sep 17 18:15:05.345923 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.345894    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07de62a1-95ce-4844-ae9b-522fb83963f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07de62a1-95ce-4844-ae9b-522fb83963f8" (UID: "07de62a1-95ce-4844-ae9b-522fb83963f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:05.346015 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.345980    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07de62a1-95ce-4844-ae9b-522fb83963f8-kube-api-access-zr5vp" (OuterVolumeSpecName: "kube-api-access-zr5vp") pod "07de62a1-95ce-4844-ae9b-522fb83963f8" (UID: "07de62a1-95ce-4844-ae9b-522fb83963f8"). InnerVolumeSpecName "kube-api-access-zr5vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:05.444529 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.444478    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr5vp\" (UniqueName: \"kubernetes.io/projected/07de62a1-95ce-4844-ae9b-522fb83963f8-kube-api-access-zr5vp\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:05.444529 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.444540    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07de62a1-95ce-4844-ae9b-522fb83963f8-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:05.608771 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.608737    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"]
Sep 17 18:15:05.612336 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.612300    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-285x5"]
Sep 17 18:15:05.938869 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:05.938780    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07de62a1-95ce-4844-ae9b-522fb83963f8" path="/var/lib/kubelet/pods/07de62a1-95ce-4844-ae9b-522fb83963f8/volumes"
Sep 17 18:15:09.852257 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:09.852224    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:09.951507 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:09.951472    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:10.054407 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.054370    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:10.105625 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.105536    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"]
Sep 17 18:15:10.105988 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.105970    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07de62a1-95ce-4844-ae9b-522fb83963f8" containerName="main"
Sep 17 18:15:10.106075 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.105992    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="07de62a1-95ce-4844-ae9b-522fb83963f8" containerName="main"
Sep 17 18:15:10.106430 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.106116    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="07de62a1-95ce-4844-ae9b-522fb83963f8" containerName="main"
Sep 17 18:15:10.111228 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.111202    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.113814 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.113723    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 17 18:15:10.113814 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.113748    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:15:10.113814 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.113772    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-fhwpl\""
Sep 17 18:15:10.114090 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.113851    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 17 18:15:10.120811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.120786    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"]
Sep 17 18:15:10.137951 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.137905    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:10.203071 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.203026    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:10.282853 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.282818    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.282853 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.282852    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.283073 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.282879    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.283073 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.282989    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.283073 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.283018    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.283073 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.283060    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.283210 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.283124    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.283210 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.283141    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4586d17b-250f-4006-9e59-f49f95ce9c36-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.283210 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.283188    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlbws\" (UniqueName: \"kubernetes.io/projected/4586d17b-250f-4006-9e59-f49f95ce9c36-kube-api-access-rlbws\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.383699 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.383612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.383699 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.383647    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4586d17b-250f-4006-9e59-f49f95ce9c36-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.383699 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.383681    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlbws\" (UniqueName: \"kubernetes.io/projected/4586d17b-250f-4006-9e59-f49f95ce9c36-kube-api-access-rlbws\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.383973 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.383715    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.383973 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.383740    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.383973 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.383779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.383973 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.383819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.383973 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.383839    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.383973 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.383883    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.384268 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.384057    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.384321 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.384286    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.384359 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.384328    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.384394 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.384340    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.384505 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.384483    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4586d17b-250f-4006-9e59-f49f95ce9c36-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.386498 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.386478    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.386589 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.386504    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.391582 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.391561    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.391680 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.391568    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlbws\" (UniqueName: \"kubernetes.io/projected/4586d17b-250f-4006-9e59-f49f95ce9c36-kube-api-access-rlbws\") pod \"router-gateway-1-openshift-default-6b46477c5b-7nvzx\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.423842 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.423806    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:10.706743 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.706681    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"]
Sep 17 18:15:10.709126 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:15:10.709091    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4586d17b_250f_4006_9e59_f49f95ce9c36.slice/crio-7cdec44ff8e7de24a735ebc684628a4f06a1f1559ad8fb59d0d89db0852399f5 WatchSource:0}: Error finding container 7cdec44ff8e7de24a735ebc684628a4f06a1f1559ad8fb59d0d89db0852399f5: Status 404 returned error can't find the container with id 7cdec44ff8e7de24a735ebc684628a4f06a1f1559ad8fb59d0d89db0852399f5
Sep 17 18:15:10.711238 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:10.711215    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:15:11.328650 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:11.328612    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx" event={"ID":"4586d17b-250f-4006-9e59-f49f95ce9c36","Type":"ContainerStarted","Data":"7cdec44ff8e7de24a735ebc684628a4f06a1f1559ad8fb59d0d89db0852399f5"}
Sep 17 18:15:13.250417 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:13.250378    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 18:15:13.250817 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:13.250465    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 18:15:13.250817 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:13.250509    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 17 18:15:13.350401 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:13.350351    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx" event={"ID":"4586d17b-250f-4006-9e59-f49f95ce9c36","Type":"ContainerStarted","Data":"19b11e23f68f9e5ccd539c5d7fca98e36db56c4b9b79d90efb4ce2b870d363f2"}
Sep 17 18:15:13.370674 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:13.370616    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx" podStartSLOduration=0.831792643 podStartE2EDuration="3.370597343s" totalImagesPullingTime="2.5388047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:13.369643178 +0000 UTC m=+1083.670465495" watchObservedRunningTime="2026-09-17 18:15:13.370597343 +0000 UTC m=+1083.671419666"
Sep 17 18:15:13.424849 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:13.424597    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:13.438443 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:13.438402    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:14.429634 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:14.429598    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:15.361527 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:15.361405    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:15.362413 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:15.362393    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:15.875587 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:15.875555    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:16.460687 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:16.460629    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:18.716099 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:18.716062    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:22.553146 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:22.553105    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:22.625193 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:22.625155    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:26.656532 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.656483    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"]
Sep 17 18:15:26.660013 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.659992    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.663203 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.663182    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-z594t\""
Sep 17 18:15:26.663330 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.663203    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 17 18:15:26.669150 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.669124    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"]
Sep 17 18:15:26.687987 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.687952    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:26.716455 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.716419    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e8f945-6a1c-4fce-a102-c33956581e42-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.716644 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.716460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.716644 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.716541    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.716644 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.716590    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.716644 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.716623    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f89nr\" (UniqueName: \"kubernetes.io/projected/99e8f945-6a1c-4fce-a102-c33956581e42-kube-api-access-f89nr\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.716782 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.716659    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.817489 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.817456    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.817692 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.817540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e8f945-6a1c-4fce-a102-c33956581e42-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.817692 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.817579    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.817692 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.817631    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.817692 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.817666    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.817899 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.817704    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f89nr\" (UniqueName: \"kubernetes.io/projected/99e8f945-6a1c-4fce-a102-c33956581e42-kube-api-access-f89nr\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.817983 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.817943    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.817983 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.817960    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.818116 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.818019    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.820156 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.820132    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.820323 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.820305    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e8f945-6a1c-4fce-a102-c33956581e42-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.825319 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.825299    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f89nr\" (UniqueName: \"kubernetes.io/projected/99e8f945-6a1c-4fce-a102-c33956581e42-kube-api-access-f89nr\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:26.971877 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:26.971777    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:27.248066 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:27.248015    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"]
Sep 17 18:15:27.250255 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:15:27.250220    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e8f945_6a1c_4fce_a102_c33956581e42.slice/crio-c80b0ebb7a47cd488a94569f5da8f7256a445b963640903a8ade6fe63dc85869 WatchSource:0}: Error finding container c80b0ebb7a47cd488a94569f5da8f7256a445b963640903a8ade6fe63dc85869: Status 404 returned error can't find the container with id c80b0ebb7a47cd488a94569f5da8f7256a445b963640903a8ade6fe63dc85869
Sep 17 18:15:27.428335 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:27.428295    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz" event={"ID":"99e8f945-6a1c-4fce-a102-c33956581e42","Type":"ContainerStarted","Data":"53e5c38ccaeee3e31ef2c4768def4048fae5ecaff931dd3e841a503e87e2bd4e"}
Sep 17 18:15:27.428524 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:27.428352    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz" event={"ID":"99e8f945-6a1c-4fce-a102-c33956581e42","Type":"ContainerStarted","Data":"c80b0ebb7a47cd488a94569f5da8f7256a445b963640903a8ade6fe63dc85869"}
Sep 17 18:15:27.446916 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:27.446866    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz" podStartSLOduration=1.446837212 podStartE2EDuration="1.446837212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:27.445183429 +0000 UTC m=+1097.746005754" watchObservedRunningTime="2026-09-17 18:15:27.446837212 +0000 UTC m=+1097.747659536"
Sep 17 18:15:27.972605 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:27.972574    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:29.209903 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:29.209828    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:29.311819 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:29.311785    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:29.408438 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:29.408402    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:29.525846 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:29.525814    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:29.592396 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:29.592361    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:33.469260 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:33.469226    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:34.187849 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.187746    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"]
Sep 17 18:15:34.193091 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.193058    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"]
Sep 17 18:15:34.193291 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.193268    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx" podUID="4586d17b-250f-4006-9e59-f49f95ce9c36" containerName="istio-proxy" containerID="cri-o://19b11e23f68f9e5ccd539c5d7fca98e36db56c4b9b79d90efb4ce2b870d363f2" gracePeriod=30
Sep 17 18:15:34.197742 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:15:34.197707    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:15:34.197742 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:15:34.197742 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:15:34.197742 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="53e5c38ccaeee3e31ef2c4768def4048fae5ecaff931dd3e841a503e87e2bd4e" cmd=["/bin/sleep","15"]
Sep 17 18:15:34.198002 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.197761    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz" message="" err=<
Sep 17 18:15:34.198002 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:15:34.198002 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:15:34.198002 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 18:15:34.198002 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:15:34.197773    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:15:34.198002 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:15:34.198002 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:15:34.198002 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz" podUID="99e8f945-6a1c-4fce-a102-c33956581e42" containerName="main" containerID="cri-o://53e5c38ccaeee3e31ef2c4768def4048fae5ecaff931dd3e841a503e87e2bd4e"
Sep 17 18:15:34.198002 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.197818    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz" podUID="99e8f945-6a1c-4fce-a102-c33956581e42" containerName="main" containerID="cri-o://53e5c38ccaeee3e31ef2c4768def4048fae5ecaff931dd3e841a503e87e2bd4e" gracePeriod=60
Sep 17 18:15:34.283313 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.283273    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:34.383551 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.383495    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:34.472732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.472645    2574 generic.go:356] "Generic (PLEG): container finished" podID="99e8f945-6a1c-4fce-a102-c33956581e42" containerID="53e5c38ccaeee3e31ef2c4768def4048fae5ecaff931dd3e841a503e87e2bd4e" exitCode=0
Sep 17 18:15:34.473090 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.472723    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz" event={"ID":"99e8f945-6a1c-4fce-a102-c33956581e42","Type":"ContainerDied","Data":"53e5c38ccaeee3e31ef2c4768def4048fae5ecaff931dd3e841a503e87e2bd4e"}
Sep 17 18:15:34.968418 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.968389    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:34.986613 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.986566    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-dshm\") pod \"99e8f945-6a1c-4fce-a102-c33956581e42\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") "
Sep 17 18:15:34.986795 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.986623    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f89nr\" (UniqueName: \"kubernetes.io/projected/99e8f945-6a1c-4fce-a102-c33956581e42-kube-api-access-f89nr\") pod \"99e8f945-6a1c-4fce-a102-c33956581e42\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") "
Sep 17 18:15:34.986795 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.986670    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-tmp-dir\") pod \"99e8f945-6a1c-4fce-a102-c33956581e42\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") "
Sep 17 18:15:34.986795 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.986762    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e8f945-6a1c-4fce-a102-c33956581e42-tls-certs\") pod \"99e8f945-6a1c-4fce-a102-c33956581e42\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") "
Sep 17 18:15:34.986795 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.986789    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-model-cache\") pod \"99e8f945-6a1c-4fce-a102-c33956581e42\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") "
Sep 17 18:15:34.987006 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.986814    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-home\") pod \"99e8f945-6a1c-4fce-a102-c33956581e42\" (UID: \"99e8f945-6a1c-4fce-a102-c33956581e42\") "
Sep 17 18:15:34.987166 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.987132    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99e8f945-6a1c-4fce-a102-c33956581e42" (UID: "99e8f945-6a1c-4fce-a102-c33956581e42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:34.987266 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.987251    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-home" (OuterVolumeSpecName: "home") pod "99e8f945-6a1c-4fce-a102-c33956581e42" (UID: "99e8f945-6a1c-4fce-a102-c33956581e42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:34.987465 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.987423    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-model-cache" (OuterVolumeSpecName: "model-cache") pod "99e8f945-6a1c-4fce-a102-c33956581e42" (UID: "99e8f945-6a1c-4fce-a102-c33956581e42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:34.989528 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.989403    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-dshm" (OuterVolumeSpecName: "dshm") pod "99e8f945-6a1c-4fce-a102-c33956581e42" (UID: "99e8f945-6a1c-4fce-a102-c33956581e42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:34.989664 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.989551    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e8f945-6a1c-4fce-a102-c33956581e42-kube-api-access-f89nr" (OuterVolumeSpecName: "kube-api-access-f89nr") pod "99e8f945-6a1c-4fce-a102-c33956581e42" (UID: "99e8f945-6a1c-4fce-a102-c33956581e42"). InnerVolumeSpecName "kube-api-access-f89nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:34.989945 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:34.989924    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e8f945-6a1c-4fce-a102-c33956581e42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99e8f945-6a1c-4fce-a102-c33956581e42" (UID: "99e8f945-6a1c-4fce-a102-c33956581e42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:35.087491 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:35.087397    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:35.087491 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:35.087431    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f89nr\" (UniqueName: \"kubernetes.io/projected/99e8f945-6a1c-4fce-a102-c33956581e42-kube-api-access-f89nr\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:35.087491 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:35.087442    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:35.087491 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:35.087451    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e8f945-6a1c-4fce-a102-c33956581e42-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:35.087491 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:35.087458    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:35.087491 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:35.087465    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e8f945-6a1c-4fce-a102-c33956581e42-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:35.481131 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:35.481050    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"
Sep 17 18:15:35.481131 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:35.481077    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz" event={"ID":"99e8f945-6a1c-4fce-a102-c33956581e42","Type":"ContainerDied","Data":"c80b0ebb7a47cd488a94569f5da8f7256a445b963640903a8ade6fe63dc85869"}
Sep 17 18:15:35.481642 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:35.481137    2574 scope.go:117] "RemoveContainer" containerID="53e5c38ccaeee3e31ef2c4768def4048fae5ecaff931dd3e841a503e87e2bd4e"
Sep 17 18:15:35.505261 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:35.505233    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"]
Sep 17 18:15:35.509047 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:35.509019    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-767kz"]
Sep 17 18:15:35.940100 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:35.940066    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99e8f945-6a1c-4fce-a102-c33956581e42" path="/var/lib/kubelet/pods/99e8f945-6a1c-4fce-a102-c33956581e42/volumes"
Sep 17 18:15:37.161482 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.161448    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:37.206548 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.206432    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-token\") pod \"4586d17b-250f-4006-9e59-f49f95ce9c36\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") "
Sep 17 18:15:37.206548 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.206477    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-podinfo\") pod \"4586d17b-250f-4006-9e59-f49f95ce9c36\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") "
Sep 17 18:15:37.206548 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.206507    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-data\") pod \"4586d17b-250f-4006-9e59-f49f95ce9c36\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") "
Sep 17 18:15:37.206843 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.206596    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlbws\" (UniqueName: \"kubernetes.io/projected/4586d17b-250f-4006-9e59-f49f95ce9c36-kube-api-access-rlbws\") pod \"4586d17b-250f-4006-9e59-f49f95ce9c36\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") "
Sep 17 18:15:37.206843 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.206628    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-credential-socket\") pod \"4586d17b-250f-4006-9e59-f49f95ce9c36\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") "
Sep 17 18:15:37.206843 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.206643    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-envoy\") pod \"4586d17b-250f-4006-9e59-f49f95ce9c36\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") "
Sep 17 18:15:37.206843 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.206678    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4586d17b-250f-4006-9e59-f49f95ce9c36-istiod-ca-cert\") pod \"4586d17b-250f-4006-9e59-f49f95ce9c36\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") "
Sep 17 18:15:37.206843 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.206703    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-workload-socket\") pod \"4586d17b-250f-4006-9e59-f49f95ce9c36\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") "
Sep 17 18:15:37.206843 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.206737    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-workload-certs\") pod \"4586d17b-250f-4006-9e59-f49f95ce9c36\" (UID: \"4586d17b-250f-4006-9e59-f49f95ce9c36\") "
Sep 17 18:15:37.207498 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.207431    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-data" (OuterVolumeSpecName: "istio-data") pod "4586d17b-250f-4006-9e59-f49f95ce9c36" (UID: "4586d17b-250f-4006-9e59-f49f95ce9c36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:37.207498 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.207447    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4586d17b-250f-4006-9e59-f49f95ce9c36" (UID: "4586d17b-250f-4006-9e59-f49f95ce9c36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:37.207729 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.207638    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4586d17b-250f-4006-9e59-f49f95ce9c36" (UID: "4586d17b-250f-4006-9e59-f49f95ce9c36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:37.207729 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.207708    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4586d17b-250f-4006-9e59-f49f95ce9c36-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4586d17b-250f-4006-9e59-f49f95ce9c36" (UID: "4586d17b-250f-4006-9e59-f49f95ce9c36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:37.207852 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.207774    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4586d17b-250f-4006-9e59-f49f95ce9c36" (UID: "4586d17b-250f-4006-9e59-f49f95ce9c36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:37.211032 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.210480    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4586d17b-250f-4006-9e59-f49f95ce9c36-kube-api-access-rlbws" (OuterVolumeSpecName: "kube-api-access-rlbws") pod "4586d17b-250f-4006-9e59-f49f95ce9c36" (UID: "4586d17b-250f-4006-9e59-f49f95ce9c36"). InnerVolumeSpecName "kube-api-access-rlbws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:37.212727 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.212664    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4586d17b-250f-4006-9e59-f49f95ce9c36" (UID: "4586d17b-250f-4006-9e59-f49f95ce9c36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:15:37.212727 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.212685    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-token" (OuterVolumeSpecName: "istio-token") pod "4586d17b-250f-4006-9e59-f49f95ce9c36" (UID: "4586d17b-250f-4006-9e59-f49f95ce9c36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:37.212899 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.212774    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4586d17b-250f-4006-9e59-f49f95ce9c36" (UID: "4586d17b-250f-4006-9e59-f49f95ce9c36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:37.308313 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.308274    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlbws\" (UniqueName: \"kubernetes.io/projected/4586d17b-250f-4006-9e59-f49f95ce9c36-kube-api-access-rlbws\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:37.308313 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.308310    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-credential-socket\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:37.308313 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.308319    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-envoy\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:37.308592 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.308328    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4586d17b-250f-4006-9e59-f49f95ce9c36-istiod-ca-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:37.308592 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.308336    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-workload-socket\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:37.308592 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.308344    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-workload-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:37.308592 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.308353    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-token\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:37.308592 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.308360    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-podinfo\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:37.308592 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.308368    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4586d17b-250f-4006-9e59-f49f95ce9c36-istio-data\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:37.493948 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.493857    2574 generic.go:356] "Generic (PLEG): container finished" podID="4586d17b-250f-4006-9e59-f49f95ce9c36" containerID="19b11e23f68f9e5ccd539c5d7fca98e36db56c4b9b79d90efb4ce2b870d363f2" exitCode=0
Sep 17 18:15:37.493948 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.493905    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"
Sep 17 18:15:37.494129 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.493945    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx" event={"ID":"4586d17b-250f-4006-9e59-f49f95ce9c36","Type":"ContainerDied","Data":"19b11e23f68f9e5ccd539c5d7fca98e36db56c4b9b79d90efb4ce2b870d363f2"}
Sep 17 18:15:37.494129 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.494021    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx" event={"ID":"4586d17b-250f-4006-9e59-f49f95ce9c36","Type":"ContainerDied","Data":"7cdec44ff8e7de24a735ebc684628a4f06a1f1559ad8fb59d0d89db0852399f5"}
Sep 17 18:15:37.494129 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.494048    2574 scope.go:117] "RemoveContainer" containerID="19b11e23f68f9e5ccd539c5d7fca98e36db56c4b9b79d90efb4ce2b870d363f2"
Sep 17 18:15:37.504405 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.504382    2574 scope.go:117] "RemoveContainer" containerID="19b11e23f68f9e5ccd539c5d7fca98e36db56c4b9b79d90efb4ce2b870d363f2"
Sep 17 18:15:37.504730 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:15:37.504709    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b11e23f68f9e5ccd539c5d7fca98e36db56c4b9b79d90efb4ce2b870d363f2\": container with ID starting with 19b11e23f68f9e5ccd539c5d7fca98e36db56c4b9b79d90efb4ce2b870d363f2 not found: ID does not exist" containerID="19b11e23f68f9e5ccd539c5d7fca98e36db56c4b9b79d90efb4ce2b870d363f2"
Sep 17 18:15:37.504817 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.504739    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b11e23f68f9e5ccd539c5d7fca98e36db56c4b9b79d90efb4ce2b870d363f2"} err="failed to get container status \"19b11e23f68f9e5ccd539c5d7fca98e36db56c4b9b79d90efb4ce2b870d363f2\": rpc error: code = NotFound desc = could not find container \"19b11e23f68f9e5ccd539c5d7fca98e36db56c4b9b79d90efb4ce2b870d363f2\": container with ID starting with 19b11e23f68f9e5ccd539c5d7fca98e36db56c4b9b79d90efb4ce2b870d363f2 not found: ID does not exist"
Sep 17 18:15:37.529135 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.529103    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"]
Sep 17 18:15:37.534973 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.534946    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7nvzx"]
Sep 17 18:15:37.938174 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:37.938137    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4586d17b-250f-4006-9e59-f49f95ce9c36" path="/var/lib/kubelet/pods/4586d17b-250f-4006-9e59-f49f95ce9c36/volumes"
Sep 17 18:15:39.330732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:39.330696    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:40.683160 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:40.683115    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:45.704919 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:45.704873    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:46.471715 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:46.471679    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:46.572588 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:46.572553    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:50.557536 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.557412    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"]
Sep 17 18:15:50.558131 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.558047    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e8f945-6a1c-4fce-a102-c33956581e42" containerName="main"
Sep 17 18:15:50.558131 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.558066    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e8f945-6a1c-4fce-a102-c33956581e42" containerName="main"
Sep 17 18:15:50.558131 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.558085    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4586d17b-250f-4006-9e59-f49f95ce9c36" containerName="istio-proxy"
Sep 17 18:15:50.558131 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.558090    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4586d17b-250f-4006-9e59-f49f95ce9c36" containerName="istio-proxy"
Sep 17 18:15:50.558301 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.558176    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4586d17b-250f-4006-9e59-f49f95ce9c36" containerName="istio-proxy"
Sep 17 18:15:50.558301 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.558188    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e8f945-6a1c-4fce-a102-c33956581e42" containerName="main"
Sep 17 18:15:50.567188 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.567161    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.568497 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.568465    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"]
Sep 17 18:15:50.569924 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.569891    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 17 18:15:50.570676 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.570654    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 18:15:50.570906 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.570752    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 18:15:50.571363 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.571343    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-5kg26\""
Sep 17 18:15:50.593979 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.593951    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:50.620287 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.620239    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpz5f\" (UniqueName: \"kubernetes.io/projected/0ae9c889-375b-486e-a766-1b4652db3a3d-kube-api-access-kpz5f\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.620454 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.620296    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.620454 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.620380    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.620454 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.620415    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.620454 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.620434    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae9c889-375b-486e-a766-1b4652db3a3d-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.620647 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.620501    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.721948 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.721903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.721948 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.721950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.722210 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.721968    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae9c889-375b-486e-a766-1b4652db3a3d-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.722210 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.722007    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.722210 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.722061    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpz5f\" (UniqueName: \"kubernetes.io/projected/0ae9c889-375b-486e-a766-1b4652db3a3d-kube-api-access-kpz5f\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.722210 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.722106    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.722453 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.722427    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.722545 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.722459    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.722545 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.722504    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.724583 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.724564    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.724760 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.724745    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae9c889-375b-486e-a766-1b4652db3a3d-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.730096 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.730073    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpz5f\" (UniqueName: \"kubernetes.io/projected/0ae9c889-375b-486e-a766-1b4652db3a3d-kube-api-access-kpz5f\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:50.880312 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:50.880214    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:51.157801 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:51.157773    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"]
Sep 17 18:15:51.159817 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:15:51.159788    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae9c889_375b_486e_a766_1b4652db3a3d.slice/crio-b79d22435fd22c114a63c62e0ba805e7ef4ad134f882eb03f7882c79ce58a7e6 WatchSource:0}: Error finding container b79d22435fd22c114a63c62e0ba805e7ef4ad134f882eb03f7882c79ce58a7e6: Status 404 returned error can't find the container with id b79d22435fd22c114a63c62e0ba805e7ef4ad134f882eb03f7882c79ce58a7e6
Sep 17 18:15:51.576369 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:51.576329    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6" event={"ID":"0ae9c889-375b-486e-a766-1b4652db3a3d","Type":"ContainerStarted","Data":"ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533"}
Sep 17 18:15:51.576786 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:51.576384    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6" event={"ID":"0ae9c889-375b-486e-a766-1b4652db3a3d","Type":"ContainerStarted","Data":"b79d22435fd22c114a63c62e0ba805e7ef4ad134f882eb03f7882c79ce58a7e6"}
Sep 17 18:15:51.601584 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:51.601492    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6" podStartSLOduration=1.601467424 podStartE2EDuration="1.601467424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:51.600272869 +0000 UTC m=+1121.901095207" watchObservedRunningTime="2026-09-17 18:15:51.601467424 +0000 UTC m=+1121.902289749"
Sep 17 18:15:51.881079 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:51.880985    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:53.183759 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:53.183720    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:53.284553 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:53.284503    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:53.385993 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:53.385958    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:54.780866 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:54.780830    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"]
Sep 17 18:15:54.791078 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:15:54.791042    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:15:54.791078 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:15:54.791078 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:15:54.791078 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533" cmd=["/bin/sleep","15"]
Sep 17 18:15:54.791348 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:54.791095    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6" message="" err=<
Sep 17 18:15:54.791348 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:15:54.791348 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:15:54.791348 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 18:15:54.791348 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:15:54.791109    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:15:54.791348 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:15:54.791348 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:15:54.791348 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6" podUID="0ae9c889-375b-486e-a766-1b4652db3a3d" containerName="main" containerID="cri-o://ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533"
Sep 17 18:15:54.791348 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:54.791147    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6" podUID="0ae9c889-375b-486e-a766-1b4652db3a3d" containerName="main" containerID="cri-o://ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533" gracePeriod=60
Sep 17 18:15:54.800987 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:54.800961    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:55.051453 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.051424    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:55.161176 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.161131    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae9c889-375b-486e-a766-1b4652db3a3d-tls-certs\") pod \"0ae9c889-375b-486e-a766-1b4652db3a3d\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") "
Sep 17 18:15:55.161387 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.161198    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-dshm\") pod \"0ae9c889-375b-486e-a766-1b4652db3a3d\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") "
Sep 17 18:15:55.161387 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.161243    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-tmp-dir\") pod \"0ae9c889-375b-486e-a766-1b4652db3a3d\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") "
Sep 17 18:15:55.161387 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.161286    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-model-cache\") pod \"0ae9c889-375b-486e-a766-1b4652db3a3d\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") "
Sep 17 18:15:55.161387 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.161314    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpz5f\" (UniqueName: \"kubernetes.io/projected/0ae9c889-375b-486e-a766-1b4652db3a3d-kube-api-access-kpz5f\") pod \"0ae9c889-375b-486e-a766-1b4652db3a3d\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") "
Sep 17 18:15:55.161387 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.161350    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-home\") pod \"0ae9c889-375b-486e-a766-1b4652db3a3d\" (UID: \"0ae9c889-375b-486e-a766-1b4652db3a3d\") "
Sep 17 18:15:55.161668 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.161549    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ae9c889-375b-486e-a766-1b4652db3a3d" (UID: "0ae9c889-375b-486e-a766-1b4652db3a3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:55.161668 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.161575    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ae9c889-375b-486e-a766-1b4652db3a3d" (UID: "0ae9c889-375b-486e-a766-1b4652db3a3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:55.161785 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.161685    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:55.161785 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.161697    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:55.161785 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.161707    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-home" (OuterVolumeSpecName: "home") pod "0ae9c889-375b-486e-a766-1b4652db3a3d" (UID: "0ae9c889-375b-486e-a766-1b4652db3a3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:55.163815 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.163778    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae9c889-375b-486e-a766-1b4652db3a3d-kube-api-access-kpz5f" (OuterVolumeSpecName: "kube-api-access-kpz5f") pod "0ae9c889-375b-486e-a766-1b4652db3a3d" (UID: "0ae9c889-375b-486e-a766-1b4652db3a3d"). InnerVolumeSpecName "kube-api-access-kpz5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:55.163815 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.163785    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ae9c889-375b-486e-a766-1b4652db3a3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ae9c889-375b-486e-a766-1b4652db3a3d" (UID: "0ae9c889-375b-486e-a766-1b4652db3a3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:55.163962 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.163820    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-dshm" (OuterVolumeSpecName: "dshm") pod "0ae9c889-375b-486e-a766-1b4652db3a3d" (UID: "0ae9c889-375b-486e-a766-1b4652db3a3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:55.262596 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.262559    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpz5f\" (UniqueName: \"kubernetes.io/projected/0ae9c889-375b-486e-a766-1b4652db3a3d-kube-api-access-kpz5f\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:55.262596 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.262590    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:55.262596 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.262599    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae9c889-375b-486e-a766-1b4652db3a3d-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:55.262596 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.262607    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae9c889-375b-486e-a766-1b4652db3a3d-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:15:55.601146 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.601110    2574 generic.go:356] "Generic (PLEG): container finished" podID="0ae9c889-375b-486e-a766-1b4652db3a3d" containerID="ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533" exitCode=0
Sep 17 18:15:55.601311 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.601174    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"
Sep 17 18:15:55.601311 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.601204    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6" event={"ID":"0ae9c889-375b-486e-a766-1b4652db3a3d","Type":"ContainerDied","Data":"ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533"}
Sep 17 18:15:55.601311 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.601266    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6" event={"ID":"0ae9c889-375b-486e-a766-1b4652db3a3d","Type":"ContainerDied","Data":"b79d22435fd22c114a63c62e0ba805e7ef4ad134f882eb03f7882c79ce58a7e6"}
Sep 17 18:15:55.601311 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.601296    2574 scope.go:117] "RemoveContainer" containerID="ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533"
Sep 17 18:15:55.611884 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.611861    2574 scope.go:117] "RemoveContainer" containerID="ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533"
Sep 17 18:15:55.612163 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:15:55.612144    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533\": container with ID starting with ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533 not found: ID does not exist" containerID="ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533"
Sep 17 18:15:55.612212 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.612172    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533"} err="failed to get container status \"ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533\": rpc error: code = NotFound desc = could not find container \"ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533\": container with ID starting with ef58425331b1eee763ba44f49d1271ff1a91d1b0792363755267a3d66bca3533 not found: ID does not exist"
Sep 17 18:15:55.626386 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.626361    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"]
Sep 17 18:15:55.630672 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.630648    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwqnq6"]
Sep 17 18:15:55.938294 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:55.938203    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ae9c889-375b-486e-a766-1b4652db3a3d" path="/var/lib/kubelet/pods/0ae9c889-375b-486e-a766-1b4652db3a3d/volumes"
Sep 17 18:15:58.272744 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:15:58.272710    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:02.974171 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:02.974098    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"]
Sep 17 18:16:02.974774 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:02.974750    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ae9c889-375b-486e-a766-1b4652db3a3d" containerName="main"
Sep 17 18:16:02.974934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:02.974915    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ae9c889-375b-486e-a766-1b4652db3a3d" containerName="main"
Sep 17 18:16:02.975069 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:02.975056    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ae9c889-375b-486e-a766-1b4652db3a3d" containerName="main"
Sep 17 18:16:02.978369 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:02.978346    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:02.981303 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:02.981280    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:16:02.981448 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:02.981280    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 17 18:16:02.981448 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:02.981280    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-p6n8x\""
Sep 17 18:16:02.981602 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:02.981280    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 18:16:02.981602 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:02.981285    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 18:16:02.981602 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:02.981345    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 18:16:02.991821 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:02.991792    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"]
Sep 17 18:16:03.009013 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.008983    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:03.136277 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.136239    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-dshm\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.136432 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.136290    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-tmp-dir\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.136432 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.136341    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f729af8-9cd2-40dc-961f-11ae63801569-tls-certs\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.136432 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.136418    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f729af8-9cd2-40dc-961f-11ae63801569-cabundle-cert\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.136588 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.136438    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-home\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.136588 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.136491    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-model-cache\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.136588 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.136544    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-kserve-provision-location\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.136588 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.136581    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25kk7\" (UniqueName: \"kubernetes.io/projected/6f729af8-9cd2-40dc-961f-11ae63801569-kube-api-access-25kk7\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.237441 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.237349    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-tmp-dir\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.237441 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.237384    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f729af8-9cd2-40dc-961f-11ae63801569-tls-certs\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.237441 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.237432    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f729af8-9cd2-40dc-961f-11ae63801569-cabundle-cert\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.237767 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.237451    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-home\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.237767 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.237475    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-model-cache\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.237767 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.237493    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-kserve-provision-location\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.237767 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.237674    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25kk7\" (UniqueName: \"kubernetes.io/projected/6f729af8-9cd2-40dc-961f-11ae63801569-kube-api-access-25kk7\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.237767 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.237725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-dshm\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.238007 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.237832    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-tmp-dir\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.238007 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.237877    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-home\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.238007 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.237901    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-kserve-provision-location\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.238007 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.237985    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-model-cache\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.238337 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.238316    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f729af8-9cd2-40dc-961f-11ae63801569-cabundle-cert\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.240158 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.240133    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-dshm\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.240428 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.240412    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f729af8-9cd2-40dc-961f-11ae63801569-tls-certs\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.246396 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.246377    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25kk7\" (UniqueName: \"kubernetes.io/projected/6f729af8-9cd2-40dc-961f-11ae63801569-kube-api-access-25kk7\") pod \"auth-enabled-test-kserve-68c55859dc-j7hqn\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.292714 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.292677    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:03.364545 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.364446    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:03.553712 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.553676    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:03.578135 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.578113    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"]
Sep 17 18:16:03.580552 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:16:03.580488    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f729af8_9cd2_40dc_961f_11ae63801569.slice/crio-e261c2d5c41d5e4dc34310012544a4e50d6961b267a767f52a6e5777af2d48c1 WatchSource:0}: Error finding container e261c2d5c41d5e4dc34310012544a4e50d6961b267a767f52a6e5777af2d48c1: Status 404 returned error can't find the container with id e261c2d5c41d5e4dc34310012544a4e50d6961b267a767f52a6e5777af2d48c1
Sep 17 18:16:03.655209 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.655177    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" event={"ID":"6f729af8-9cd2-40dc-961f-11ae63801569","Type":"ContainerStarted","Data":"ff48ea8be22dffc0110a65d0cd1ad2757251abb2b4224e125b62a427dce03067"}
Sep 17 18:16:03.655357 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:03.655235    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" event={"ID":"6f729af8-9cd2-40dc-961f-11ae63801569","Type":"ContainerStarted","Data":"e261c2d5c41d5e4dc34310012544a4e50d6961b267a767f52a6e5777af2d48c1"}
Sep 17 18:16:05.382877 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:05.382842    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:06.434975 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:06.434941    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:06.676280 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:06.676244    2574 generic.go:356] "Generic (PLEG): container finished" podID="6f729af8-9cd2-40dc-961f-11ae63801569" containerID="ff48ea8be22dffc0110a65d0cd1ad2757251abb2b4224e125b62a427dce03067" exitCode=0
Sep 17 18:16:06.676428 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:06.676308    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" event={"ID":"6f729af8-9cd2-40dc-961f-11ae63801569","Type":"ContainerDied","Data":"ff48ea8be22dffc0110a65d0cd1ad2757251abb2b4224e125b62a427dce03067"}
Sep 17 18:16:09.892306 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:09.892251    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:11.086749 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:11.086705    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:11.194445 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:11.194410    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:20.629400 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:20.629305    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:35.906217 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:35.906171    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" event={"ID":"6f729af8-9cd2-40dc-961f-11ae63801569","Type":"ContainerStarted","Data":"23fbaf70f569eacaea0beb81f31c960c8cd4b618af3d2a2e0cdd931bbabfbe30"}
Sep 17 18:16:35.930403 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:35.930334    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" podStartSLOduration=3.213111392 podStartE2EDuration="33.930311153s" totalImagesPullingTime="28.442232088s" totalInitContainerRuntime="2.274967673s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:35.927822739 +0000 UTC m=+1166.228645064" watchObservedRunningTime="2026-09-17 18:16:35.930311153 +0000 UTC m=+1166.231133478"
Sep 17 18:16:36.292843 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:36.292800    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:40.384006 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.383973    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:40.390670 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.390644    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"]
Sep 17 18:16:40.401547 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:16:40.401481    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:16:40.401547 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:16:40.401547 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:16:40.401547 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2" cmd=["/bin/sleep","15"]
Sep 17 18:16:40.401866 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.401575    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p" message="" err=<
Sep 17 18:16:40.401866 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:16:40.401866 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:16:40.401866 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 18:16:40.401866 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:16:40.401596    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:16:40.401866 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:16:40.401866 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:16:40.401866 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p" podUID="8dab5c28-7745-4e3c-a45e-a7a831d44ef6" containerName="main" containerID="cri-o://1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2"
Sep 17 18:16:40.401866 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.401643    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p" podUID="8dab5c28-7745-4e3c-a45e-a7a831d44ef6" containerName="main" containerID="cri-o://1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2" gracePeriod=60
Sep 17 18:16:40.642103 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.641996    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:40.793458 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.793424    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:16:40.910773 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.910684    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-model-cache\") pod \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") "
Sep 17 18:16:40.910773 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.910745    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-dshm\") pod \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") "
Sep 17 18:16:40.910987 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.910794    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-tls-certs\") pod \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") "
Sep 17 18:16:40.910987 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.910845    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-tmp-dir\") pod \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") "
Sep 17 18:16:40.910987 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.910929    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6sb8\" (UniqueName: \"kubernetes.io/projected/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-kube-api-access-g6sb8\") pod \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") "
Sep 17 18:16:40.910987 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.910960    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-home\") pod \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\" (UID: \"8dab5c28-7745-4e3c-a45e-a7a831d44ef6\") "
Sep 17 18:16:40.911202 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.910982    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-model-cache" (OuterVolumeSpecName: "model-cache") pod "8dab5c28-7745-4e3c-a45e-a7a831d44ef6" (UID: "8dab5c28-7745-4e3c-a45e-a7a831d44ef6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:40.911202 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.911180    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:16:40.911202 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.911175    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8dab5c28-7745-4e3c-a45e-a7a831d44ef6" (UID: "8dab5c28-7745-4e3c-a45e-a7a831d44ef6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:40.911374 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.911318    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-home" (OuterVolumeSpecName: "home") pod "8dab5c28-7745-4e3c-a45e-a7a831d44ef6" (UID: "8dab5c28-7745-4e3c-a45e-a7a831d44ef6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:40.913374 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.913340    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-dshm" (OuterVolumeSpecName: "dshm") pod "8dab5c28-7745-4e3c-a45e-a7a831d44ef6" (UID: "8dab5c28-7745-4e3c-a45e-a7a831d44ef6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:40.913374 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.913364    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8dab5c28-7745-4e3c-a45e-a7a831d44ef6" (UID: "8dab5c28-7745-4e3c-a45e-a7a831d44ef6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:16:40.913584 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.913411    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-kube-api-access-g6sb8" (OuterVolumeSpecName: "kube-api-access-g6sb8") pod "8dab5c28-7745-4e3c-a45e-a7a831d44ef6" (UID: "8dab5c28-7745-4e3c-a45e-a7a831d44ef6"). InnerVolumeSpecName "kube-api-access-g6sb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:16:40.938116 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.938080    2574 generic.go:356] "Generic (PLEG): container finished" podID="8dab5c28-7745-4e3c-a45e-a7a831d44ef6" containerID="1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2" exitCode=0
Sep 17 18:16:40.938312 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.938137    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"
Sep 17 18:16:40.938312 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.938164    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p" event={"ID":"8dab5c28-7745-4e3c-a45e-a7a831d44ef6","Type":"ContainerDied","Data":"1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2"}
Sep 17 18:16:40.938312 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.938224    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p" event={"ID":"8dab5c28-7745-4e3c-a45e-a7a831d44ef6","Type":"ContainerDied","Data":"27e987e2691be90906b8f4d2c1e1eeee4a7bd02c3b4cd0c92a88f01f87edc65f"}
Sep 17 18:16:40.938312 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.938256    2574 scope.go:117] "RemoveContainer" containerID="1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2"
Sep 17 18:16:40.957435 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.957407    2574 scope.go:117] "RemoveContainer" containerID="1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2"
Sep 17 18:16:40.957843 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:16:40.957814    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2\": container with ID starting with 1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2 not found: ID does not exist" containerID="1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2"
Sep 17 18:16:40.957917 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.957856    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2"} err="failed to get container status \"1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2\": rpc error: code = NotFound desc = could not find container \"1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2\": container with ID starting with 1de9fbce3d6e7a07030078e277504430ab19be4839f55bec3365a598fee0d7e2 not found: ID does not exist"
Sep 17 18:16:40.965203 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.965173    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"]
Sep 17 18:16:40.968529 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:40.968487    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-t6f4p"]
Sep 17 18:16:41.011662 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:41.011623    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:16:41.011835 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:41.011673    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:16:41.011835 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:41.011684    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:16:41.011835 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:41.011694    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6sb8\" (UniqueName: \"kubernetes.io/projected/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-kube-api-access-g6sb8\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:16:41.011835 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:41.011704    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dab5c28-7745-4e3c-a45e-a7a831d44ef6-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:16:41.939140 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:41.939095    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dab5c28-7745-4e3c-a45e-a7a831d44ef6" path="/var/lib/kubelet/pods/8dab5c28-7745-4e3c-a45e-a7a831d44ef6/volumes"
Sep 17 18:16:43.292839 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:43.292795    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:16:43.294715 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:43.294684    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:16:53.294023 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:16:53.293974    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:17:03.293565 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:03.293496    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:17:11.857247 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:11.857218    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:17:11.862351 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:11.862328    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:17:13.293327 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:13.293279    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:17:23.293483 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:23.293431    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:17:30.401624 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:30.401579    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:30.496082 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:30.496040    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:33.293630 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:33.293586    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:17:37.117413 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:37.117374    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:38.619178 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.619129    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"]
Sep 17 18:17:38.619789 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.619764    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dab5c28-7745-4e3c-a45e-a7a831d44ef6" containerName="main"
Sep 17 18:17:38.619851 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.619795    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dab5c28-7745-4e3c-a45e-a7a831d44ef6" containerName="main"
Sep 17 18:17:38.619961 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.619946    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dab5c28-7745-4e3c-a45e-a7a831d44ef6" containerName="main"
Sep 17 18:17:38.622873 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.622844    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.625322 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.625291    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 18:17:38.626186 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.626163    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 18:17:38.626464 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.626189    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-7ptzw\""
Sep 17 18:17:38.626616 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.626219    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 18:17:38.629536 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.629488    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"]
Sep 17 18:17:38.647951 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.647914    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:38.736178 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.736128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.736362 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.736184    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.736362 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.736252    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.736362 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.736317    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.736362 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.736344    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fb8430-f92b-4903-9d24-8e97cccff8bb-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.736500 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.736384    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dhqr\" (UniqueName: \"kubernetes.io/projected/47fb8430-f92b-4903-9d24-8e97cccff8bb-kube-api-access-9dhqr\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.837128 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.837094    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.837128 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.837130    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.837395 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.837156    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.837395 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.837208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.837395 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.837252    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fb8430-f92b-4903-9d24-8e97cccff8bb-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.837395 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.837287    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dhqr\" (UniqueName: \"kubernetes.io/projected/47fb8430-f92b-4903-9d24-8e97cccff8bb-kube-api-access-9dhqr\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.837670 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.837649    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.837725 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.837698    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.837784 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.837731    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.839751 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.839720    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.839975 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.839959    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fb8430-f92b-4903-9d24-8e97cccff8bb-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.847856 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.847828    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dhqr\" (UniqueName: \"kubernetes.io/projected/47fb8430-f92b-4903-9d24-8e97cccff8bb-kube-api-access-9dhqr\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:38.885161 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.885074    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:38.935912 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:38.935876    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:39.211662 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:39.211632    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"]
Sep 17 18:17:39.213443 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:17:39.213401    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fb8430_f92b_4903_9d24_8e97cccff8bb.slice/crio-a9306a7aedf52078b0168813f24458468348a59b236097942b94796cf6e6fc1a WatchSource:0}: Error finding container a9306a7aedf52078b0168813f24458468348a59b236097942b94796cf6e6fc1a: Status 404 returned error can't find the container with id a9306a7aedf52078b0168813f24458468348a59b236097942b94796cf6e6fc1a
Sep 17 18:17:39.272785 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:39.272752    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn" event={"ID":"47fb8430-f92b-4903-9d24-8e97cccff8bb","Type":"ContainerStarted","Data":"a9306a7aedf52078b0168813f24458468348a59b236097942b94796cf6e6fc1a"}
Sep 17 18:17:40.281374 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:40.281330    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn" event={"ID":"47fb8430-f92b-4903-9d24-8e97cccff8bb","Type":"ContainerStarted","Data":"995597542a877e5ebe2f4f53941170ec8c734f623f57206f8d6eca1da0ac14f8"}
Sep 17 18:17:40.300150 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:40.300084    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn" podStartSLOduration=2.300060894 podStartE2EDuration="2.300060894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:40.297342686 +0000 UTC m=+1230.598165015" watchObservedRunningTime="2026-09-17 18:17:40.300060894 +0000 UTC m=+1230.600883219"
Sep 17 18:17:40.936352 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:40.936305    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:43.293588 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:43.293535    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:17:43.410142 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:43.410107    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:43.480499 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:43.480458    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:48.936828 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:48.936721    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:48.950120 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:48.950093    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:49.351726 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:49.351690    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:17:53.293385 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:17:53.293280    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:18:03.293726 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:03.293676    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:18:11.250707 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:11.250668    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:13.293384 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:13.293336    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:18:16.219785 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.219742    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:16.224815 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.224785    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"]
Sep 17 18:18:16.237193 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:18:16.237156    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:18:16.237193 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:18:16.237193 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:18:16.237193 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="995597542a877e5ebe2f4f53941170ec8c734f623f57206f8d6eca1da0ac14f8" cmd=["/bin/sleep","15"]
Sep 17 18:18:16.237558 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.237218    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn" message="" err=<
Sep 17 18:18:16.237558 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:18:16.237558 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:18:16.237558 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 18:18:16.237558 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:18:16.237234    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:18:16.237558 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:18:16.237558 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:18:16.237558 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn" podUID="47fb8430-f92b-4903-9d24-8e97cccff8bb" containerName="main" containerID="cri-o://995597542a877e5ebe2f4f53941170ec8c734f623f57206f8d6eca1da0ac14f8"
Sep 17 18:18:16.237558 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.237293    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn" podUID="47fb8430-f92b-4903-9d24-8e97cccff8bb" containerName="main" containerID="cri-o://995597542a877e5ebe2f4f53941170ec8c734f623f57206f8d6eca1da0ac14f8" gracePeriod=60
Sep 17 18:18:16.304743 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.304703    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:16.335642 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.335597    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn" podUID="47fb8430-f92b-4903-9d24-8e97cccff8bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 18:18:16.494133 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.494035    2574 generic.go:356] "Generic (PLEG): container finished" podID="47fb8430-f92b-4903-9d24-8e97cccff8bb" containerID="995597542a877e5ebe2f4f53941170ec8c734f623f57206f8d6eca1da0ac14f8" exitCode=0
Sep 17 18:18:16.494299 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.494126    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn" event={"ID":"47fb8430-f92b-4903-9d24-8e97cccff8bb","Type":"ContainerDied","Data":"995597542a877e5ebe2f4f53941170ec8c734f623f57206f8d6eca1da0ac14f8"}
Sep 17 18:18:16.590687 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.590662    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:18:16.684745 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.684709    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-model-cache\") pod \"47fb8430-f92b-4903-9d24-8e97cccff8bb\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") "
Sep 17 18:18:16.684944 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.684762    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-dshm\") pod \"47fb8430-f92b-4903-9d24-8e97cccff8bb\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") "
Sep 17 18:18:16.684944 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.684810    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-tmp-dir\") pod \"47fb8430-f92b-4903-9d24-8e97cccff8bb\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") "
Sep 17 18:18:16.684944 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.684847    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fb8430-f92b-4903-9d24-8e97cccff8bb-tls-certs\") pod \"47fb8430-f92b-4903-9d24-8e97cccff8bb\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") "
Sep 17 18:18:16.684944 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.684906    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dhqr\" (UniqueName: \"kubernetes.io/projected/47fb8430-f92b-4903-9d24-8e97cccff8bb-kube-api-access-9dhqr\") pod \"47fb8430-f92b-4903-9d24-8e97cccff8bb\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") "
Sep 17 18:18:16.684944 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.684942    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-home\") pod \"47fb8430-f92b-4903-9d24-8e97cccff8bb\" (UID: \"47fb8430-f92b-4903-9d24-8e97cccff8bb\") "
Sep 17 18:18:16.685227 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.684991    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "47fb8430-f92b-4903-9d24-8e97cccff8bb" (UID: "47fb8430-f92b-4903-9d24-8e97cccff8bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:16.685227 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.685023    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47fb8430-f92b-4903-9d24-8e97cccff8bb" (UID: "47fb8430-f92b-4903-9d24-8e97cccff8bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:16.685227 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.685203    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:18:16.685227 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.685220    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:18:16.685394 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.685289    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-home" (OuterVolumeSpecName: "home") pod "47fb8430-f92b-4903-9d24-8e97cccff8bb" (UID: "47fb8430-f92b-4903-9d24-8e97cccff8bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:16.687295 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.687271    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fb8430-f92b-4903-9d24-8e97cccff8bb-kube-api-access-9dhqr" (OuterVolumeSpecName: "kube-api-access-9dhqr") pod "47fb8430-f92b-4903-9d24-8e97cccff8bb" (UID: "47fb8430-f92b-4903-9d24-8e97cccff8bb"). InnerVolumeSpecName "kube-api-access-9dhqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:16.687400 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.687321    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-dshm" (OuterVolumeSpecName: "dshm") pod "47fb8430-f92b-4903-9d24-8e97cccff8bb" (UID: "47fb8430-f92b-4903-9d24-8e97cccff8bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:16.687568 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.687552    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47fb8430-f92b-4903-9d24-8e97cccff8bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47fb8430-f92b-4903-9d24-8e97cccff8bb" (UID: "47fb8430-f92b-4903-9d24-8e97cccff8bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:16.786073 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.786039    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fb8430-f92b-4903-9d24-8e97cccff8bb-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:18:16.786073 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.786068    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dhqr\" (UniqueName: \"kubernetes.io/projected/47fb8430-f92b-4903-9d24-8e97cccff8bb-kube-api-access-9dhqr\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:18:16.786073 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.786080    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:18:16.786334 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:16.786095    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47fb8430-f92b-4903-9d24-8e97cccff8bb-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:18:17.501371 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:17.501336    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"
Sep 17 18:18:17.501891 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:17.501372    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn" event={"ID":"47fb8430-f92b-4903-9d24-8e97cccff8bb","Type":"ContainerDied","Data":"a9306a7aedf52078b0168813f24458468348a59b236097942b94796cf6e6fc1a"}
Sep 17 18:18:17.501891 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:17.501455    2574 scope.go:117] "RemoveContainer" containerID="995597542a877e5ebe2f4f53941170ec8c734f623f57206f8d6eca1da0ac14f8"
Sep 17 18:18:17.526030 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:17.525997    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"]
Sep 17 18:18:17.529273 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:17.529245    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444xm5wn"]
Sep 17 18:18:17.938445 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:17.938402    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47fb8430-f92b-4903-9d24-8e97cccff8bb" path="/var/lib/kubelet/pods/47fb8430-f92b-4903-9d24-8e97cccff8bb/volumes"
Sep 17 18:18:23.293791 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:23.293739    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:18:33.303705 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:33.303662    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:18:33.312635 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:33.312600    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:18:37.915211 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:37.915170    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:37.928602 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:37.928571    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"]
Sep 17 18:18:37.982436 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:37.982404    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:45.292630 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:45.292587    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:52.953205 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:52.953154    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main" containerID="cri-o://23fbaf70f569eacaea0beb81f31c960c8cd4b618af3d2a2e0cdd931bbabfbe30" gracePeriod=45
Sep 17 18:18:53.305253 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.305207    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:18:53.705418 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.705377    2574 generic.go:356] "Generic (PLEG): container finished" podID="6f729af8-9cd2-40dc-961f-11ae63801569" containerID="23fbaf70f569eacaea0beb81f31c960c8cd4b618af3d2a2e0cdd931bbabfbe30" exitCode=0
Sep 17 18:18:53.705590 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.705447    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" event={"ID":"6f729af8-9cd2-40dc-961f-11ae63801569","Type":"ContainerDied","Data":"23fbaf70f569eacaea0beb81f31c960c8cd4b618af3d2a2e0cdd931bbabfbe30"}
Sep 17 18:18:53.705590 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.705500    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn" event={"ID":"6f729af8-9cd2-40dc-961f-11ae63801569","Type":"ContainerDied","Data":"e261c2d5c41d5e4dc34310012544a4e50d6961b267a767f52a6e5777af2d48c1"}
Sep 17 18:18:53.705590 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.705542    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e261c2d5c41d5e4dc34310012544a4e50d6961b267a767f52a6e5777af2d48c1"
Sep 17 18:18:53.708705 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.708689    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:18:53.801680 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.801644    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-kserve-provision-location\") pod \"6f729af8-9cd2-40dc-961f-11ae63801569\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") "
Sep 17 18:18:53.801680 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.801685    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-model-cache\") pod \"6f729af8-9cd2-40dc-961f-11ae63801569\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") "
Sep 17 18:18:53.801934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.801706    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-home\") pod \"6f729af8-9cd2-40dc-961f-11ae63801569\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") "
Sep 17 18:18:53.801934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.801725    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25kk7\" (UniqueName: \"kubernetes.io/projected/6f729af8-9cd2-40dc-961f-11ae63801569-kube-api-access-25kk7\") pod \"6f729af8-9cd2-40dc-961f-11ae63801569\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") "
Sep 17 18:18:53.801934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.801777    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f729af8-9cd2-40dc-961f-11ae63801569-cabundle-cert\") pod \"6f729af8-9cd2-40dc-961f-11ae63801569\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") "
Sep 17 18:18:53.801934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.801807    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-tmp-dir\") pod \"6f729af8-9cd2-40dc-961f-11ae63801569\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") "
Sep 17 18:18:53.801934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.801833    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-dshm\") pod \"6f729af8-9cd2-40dc-961f-11ae63801569\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") "
Sep 17 18:18:53.801934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.801856    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f729af8-9cd2-40dc-961f-11ae63801569-tls-certs\") pod \"6f729af8-9cd2-40dc-961f-11ae63801569\" (UID: \"6f729af8-9cd2-40dc-961f-11ae63801569\") "
Sep 17 18:18:53.802257 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.802003    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f729af8-9cd2-40dc-961f-11ae63801569" (UID: "6f729af8-9cd2-40dc-961f-11ae63801569"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:53.802257 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.802151    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:18:53.802257 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.802173    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f729af8-9cd2-40dc-961f-11ae63801569-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f729af8-9cd2-40dc-961f-11ae63801569" (UID: "6f729af8-9cd2-40dc-961f-11ae63801569"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:53.802451 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.802425    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-home" (OuterVolumeSpecName: "home") pod "6f729af8-9cd2-40dc-961f-11ae63801569" (UID: "6f729af8-9cd2-40dc-961f-11ae63801569"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:53.804405 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.804365    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-dshm" (OuterVolumeSpecName: "dshm") pod "6f729af8-9cd2-40dc-961f-11ae63801569" (UID: "6f729af8-9cd2-40dc-961f-11ae63801569"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:53.804565 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.804433    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f729af8-9cd2-40dc-961f-11ae63801569-kube-api-access-25kk7" (OuterVolumeSpecName: "kube-api-access-25kk7") pod "6f729af8-9cd2-40dc-961f-11ae63801569" (UID: "6f729af8-9cd2-40dc-961f-11ae63801569"). InnerVolumeSpecName "kube-api-access-25kk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:53.804647 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.804586    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f729af8-9cd2-40dc-961f-11ae63801569-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f729af8-9cd2-40dc-961f-11ae63801569" (UID: "6f729af8-9cd2-40dc-961f-11ae63801569"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:53.815087 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.815017    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f729af8-9cd2-40dc-961f-11ae63801569" (UID: "6f729af8-9cd2-40dc-961f-11ae63801569"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:53.822638 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.822614    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f729af8-9cd2-40dc-961f-11ae63801569" (UID: "6f729af8-9cd2-40dc-961f-11ae63801569"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:53.903011 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.902981    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:18:53.903011 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.903007    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f729af8-9cd2-40dc-961f-11ae63801569-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:18:53.903181 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.903018    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:18:53.903181 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.903028    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:18:53.903181 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.903036    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25kk7\" (UniqueName: \"kubernetes.io/projected/6f729af8-9cd2-40dc-961f-11ae63801569-kube-api-access-25kk7\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:18:53.903181 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.903044    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f729af8-9cd2-40dc-961f-11ae63801569-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:18:53.903181 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:53.903052    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f729af8-9cd2-40dc-961f-11ae63801569-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:18:54.711044 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:54.711008    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"
Sep 17 18:18:54.729887 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:54.729838    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"]
Sep 17 18:18:54.732668 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:54.732642    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68c55859dc-j7hqn"]
Sep 17 18:18:55.938153 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:18:55.938118    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" path="/var/lib/kubelet/pods/6f729af8-9cd2-40dc-961f-11ae63801569/volumes"
Sep 17 18:19:06.301959 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:06.301926    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:06.395376 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:06.395334    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:12.799992 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:12.799959    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:18.576185 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:18.576153    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:18.676632 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:18.676597    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:21.259571 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.259469    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"]
Sep 17 18:19:21.260091 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.260067    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="storage-initializer"
Sep 17 18:19:21.260176 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.260094    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="storage-initializer"
Sep 17 18:19:21.260176 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.260108    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47fb8430-f92b-4903-9d24-8e97cccff8bb" containerName="main"
Sep 17 18:19:21.260176 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.260115    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="47fb8430-f92b-4903-9d24-8e97cccff8bb" containerName="main"
Sep 17 18:19:21.260176 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.260129    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main"
Sep 17 18:19:21.260176 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.260135    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main"
Sep 17 18:19:21.260388 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.260221    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f729af8-9cd2-40dc-961f-11ae63801569" containerName="main"
Sep 17 18:19:21.260388 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.260238    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="47fb8430-f92b-4903-9d24-8e97cccff8bb" containerName="main"
Sep 17 18:19:21.263046 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.263020    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.265398 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.265372    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 18:19:21.265544 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.265375    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 18:19:21.266328 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.266305    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-nkvkf\""
Sep 17 18:19:21.266442 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.266304    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 18:19:21.266442 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.266335    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 18:19:21.266442 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.266311    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:19:21.275663 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.275636    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"]
Sep 17 18:19:21.291237 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.291196    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:21.350174 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.350135    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.350361 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.350189    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.350361 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.350215    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-dshm\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.350361 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.350249    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8522a2fc-372a-4849-9c0e-a9033437657f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.350361 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.350281    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-home\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.350361 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.350310    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8522a2fc-372a-4849-9c0e-a9033437657f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.350361 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.350330    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.350361 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.350354    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d452\" (UniqueName: \"kubernetes.io/projected/8522a2fc-372a-4849-9c0e-a9033437657f-kube-api-access-7d452\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.451001 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.450962    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.451177 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.451040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.451177 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.451060    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-dshm\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.451177 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.451084    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8522a2fc-372a-4849-9c0e-a9033437657f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.451177 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.451108    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-home\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.451439 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.451403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8522a2fc-372a-4849-9c0e-a9033437657f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.451532 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.451472    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.451598 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.451427    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.451598 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.451552    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.451598 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.451527    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d452\" (UniqueName: \"kubernetes.io/projected/8522a2fc-372a-4849-9c0e-a9033437657f-kube-api-access-7d452\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.451756 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.451717    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-home\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.451842 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.451816    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.452100 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.452070    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8522a2fc-372a-4849-9c0e-a9033437657f-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.454092 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.454061    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-dshm\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.454364 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.454341    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8522a2fc-372a-4849-9c0e-a9033437657f-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.466094 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.466060    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d452\" (UniqueName: \"kubernetes.io/projected/8522a2fc-372a-4849-9c0e-a9033437657f-kube-api-access-7d452\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.471749 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.471678    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"]
Sep 17 18:19:21.476482 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.476460    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:19:21.487712 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.487684    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"]
Sep 17 18:19:21.506886 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.506842    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:21.552505 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.552421    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14e573ca-198d-429c-8a24-4401b9b7c00b-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:19:21.552676 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.552508    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e573ca-198d-429c-8a24-4401b9b7c00b-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:19:21.552676 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.552562    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14e573ca-198d-429c-8a24-4401b9b7c00b-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:19:21.552676 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.552593    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e573ca-198d-429c-8a24-4401b9b7c00b-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:19:21.566931 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.566898    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:21.576876 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.576850    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:21.653962 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.653923    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e573ca-198d-429c-8a24-4401b9b7c00b-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:19:21.653962 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.653966    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14e573ca-198d-429c-8a24-4401b9b7c00b-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:19:21.654199 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.654012    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e573ca-198d-429c-8a24-4401b9b7c00b-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:19:21.654199 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.654033    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14e573ca-198d-429c-8a24-4401b9b7c00b-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:19:21.654570 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.654493    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e573ca-198d-429c-8a24-4401b9b7c00b-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:19:21.654666 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.654549    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14e573ca-198d-429c-8a24-4401b9b7c00b-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:19:21.655048 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.655020    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14e573ca-198d-429c-8a24-4401b9b7c00b-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:19:21.657063 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.657043    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e573ca-198d-429c-8a24-4401b9b7c00b-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:19:21.790473 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.790439    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:19:21.862570 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:21.862543    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"]
Sep 17 18:19:21.863354 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:19:21.863327    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8522a2fc_372a_4849_9c0e_a9033437657f.slice/crio-124495bb382a3e8c070997c1f8a161f2f3189a6c90e4a78aa7495d7d4d2d37c2 WatchSource:0}: Error finding container 124495bb382a3e8c070997c1f8a161f2f3189a6c90e4a78aa7495d7d4d2d37c2: Status 404 returned error can't find the container with id 124495bb382a3e8c070997c1f8a161f2f3189a6c90e4a78aa7495d7d4d2d37c2
Sep 17 18:19:22.067201 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:22.067170    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"]
Sep 17 18:19:22.068819 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:19:22.068789    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e573ca_198d_429c_8a24_4401b9b7c00b.slice/crio-4dacff86ccf55b09326079ed00ae694ec651dbde3957494c0621ab7ee44d656e WatchSource:0}: Error finding container 4dacff86ccf55b09326079ed00ae694ec651dbde3957494c0621ab7ee44d656e: Status 404 returned error can't find the container with id 4dacff86ccf55b09326079ed00ae694ec651dbde3957494c0621ab7ee44d656e
Sep 17 18:19:22.865456 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:22.865413    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn" event={"ID":"14e573ca-198d-429c-8a24-4401b9b7c00b","Type":"ContainerStarted","Data":"29637d1551063601653e88c909d4fdd32fc2389e2e34919189006602ea71bbf7"}
Sep 17 18:19:22.865947 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:22.865485    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn" event={"ID":"14e573ca-198d-429c-8a24-4401b9b7c00b","Type":"ContainerStarted","Data":"4dacff86ccf55b09326079ed00ae694ec651dbde3957494c0621ab7ee44d656e"}
Sep 17 18:19:22.867440 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:22.867407    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld" event={"ID":"8522a2fc-372a-4849-9c0e-a9033437657f","Type":"ContainerStarted","Data":"8224e2ab119fc926e25a9d8a8f3b3a531b64f64ba510ef969a7c7ff3232b5708"}
Sep 17 18:19:22.867618 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:22.867469    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld" event={"ID":"8522a2fc-372a-4849-9c0e-a9033437657f","Type":"ContainerStarted","Data":"124495bb382a3e8c070997c1f8a161f2f3189a6c90e4a78aa7495d7d4d2d37c2"}
Sep 17 18:19:24.880265 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:24.880229    2574 generic.go:356] "Generic (PLEG): container finished" podID="14e573ca-198d-429c-8a24-4401b9b7c00b" containerID="29637d1551063601653e88c909d4fdd32fc2389e2e34919189006602ea71bbf7" exitCode=0
Sep 17 18:19:24.880742 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:24.880306    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn" event={"ID":"14e573ca-198d-429c-8a24-4401b9b7c00b","Type":"ContainerDied","Data":"29637d1551063601653e88c909d4fdd32fc2389e2e34919189006602ea71bbf7"}
Sep 17 18:19:24.882177 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:24.882152    2574 generic.go:356] "Generic (PLEG): container finished" podID="8522a2fc-372a-4849-9c0e-a9033437657f" containerID="8224e2ab119fc926e25a9d8a8f3b3a531b64f64ba510ef969a7c7ff3232b5708" exitCode=0
Sep 17 18:19:24.882251 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:24.882229    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld" event={"ID":"8522a2fc-372a-4849-9c0e-a9033437657f","Type":"ContainerDied","Data":"8224e2ab119fc926e25a9d8a8f3b3a531b64f64ba510ef969a7c7ff3232b5708"}
Sep 17 18:19:25.893337 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:25.893280    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld" event={"ID":"8522a2fc-372a-4849-9c0e-a9033437657f","Type":"ContainerStarted","Data":"63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f"}
Sep 17 18:19:26.577570 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:26.577509    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:28.056882 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:28.056832    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:28.157202 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:28.157117    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:31.578014 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:31.577802    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:31.595571 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:31.595534    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:31.615806 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:31.615748    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld" podStartSLOduration=8.073946941 podStartE2EDuration="10.61573215s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.541785209s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:25.915687687 +0000 UTC m=+1336.216510013" watchObservedRunningTime="2026-09-17 18:19:31.61573215 +0000 UTC m=+1341.916554474"
Sep 17 18:19:31.961050 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:31.960770    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:19:34.320530 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:34.320433    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:34.472549 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:34.472454    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:40.169253 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:40.169213    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:40.276141 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:19:40.276091    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:06.919753 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:06.919693    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:08.204807 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:08.204738    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:12.232622 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:12.232580    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn" event={"ID":"14e573ca-198d-429c-8a24-4401b9b7c00b","Type":"ContainerStarted","Data":"5ab7a164e81ded33443d0fd63087db3d1ffe721b5fc3b0dea3f27366f9ffb069"}
Sep 17 18:20:12.233045 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:12.232829    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:20:12.249979 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:12.249919    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn" podStartSLOduration=1.599177157 podStartE2EDuration="51.249901722s" totalImagesPullingTime="47.134437122s" totalInitContainerRuntime="2.516287443s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:12.248916702 +0000 UTC m=+1382.549739027" watchObservedRunningTime="2026-09-17 18:20:12.249901722 +0000 UTC m=+1382.550724045"
Sep 17 18:20:14.677916 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:14.677875    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:43.250008 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:43.249973    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:20:44.210845 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:44.210802    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn_14e573ca-198d-429c-8a24-4401b9b7c00b/main/0.log"
Sep 17 18:20:44.926427 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:44.926390    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:44.948795 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:44.948760    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"]
Sep 17 18:20:44.959076 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:20:44.959033    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:20:44.959076 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:20:44.959076 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:20:44.959076 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f" cmd=["/bin/sleep","15"]
Sep 17 18:20:44.959441 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:44.959099    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld" message="" err=<
Sep 17 18:20:44.959441 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:20:44.959441 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:20:44.959441 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 18:20:44.959441 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:20:44.959117    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:20:44.959441 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:20:44.959441 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:20:44.959441 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld" podUID="8522a2fc-372a-4849-9c0e-a9033437657f" containerName="main" containerID="cri-o://63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f"
Sep 17 18:20:44.959441 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:44.959174    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld" podUID="8522a2fc-372a-4849-9c0e-a9033437657f" containerName="main" containerID="cri-o://63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f" gracePeriod=60
Sep 17 18:20:44.965412 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:44.965348    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"]
Sep 17 18:20:44.965620 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:44.965583    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn" podUID="14e573ca-198d-429c-8a24-4401b9b7c00b" containerName="main" containerID="cri-o://5ab7a164e81ded33443d0fd63087db3d1ffe721b5fc3b0dea3f27366f9ffb069" gracePeriod=30
Sep 17 18:20:45.006853 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.006794    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:45.372655 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.372623    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:20:45.421275 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.421234    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-kserve-provision-location\") pod \"8522a2fc-372a-4849-9c0e-a9033437657f\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") "
Sep 17 18:20:45.421453 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.421329    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8522a2fc-372a-4849-9c0e-a9033437657f-tls-certs\") pod \"8522a2fc-372a-4849-9c0e-a9033437657f\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") "
Sep 17 18:20:45.421453 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.421355    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-dshm\") pod \"8522a2fc-372a-4849-9c0e-a9033437657f\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") "
Sep 17 18:20:45.421453 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.421378    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-tmp-dir\") pod \"8522a2fc-372a-4849-9c0e-a9033437657f\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") "
Sep 17 18:20:45.421453 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.421407    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-home\") pod \"8522a2fc-372a-4849-9c0e-a9033437657f\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") "
Sep 17 18:20:45.421453 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.421430    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-model-cache\") pod \"8522a2fc-372a-4849-9c0e-a9033437657f\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") "
Sep 17 18:20:45.421768 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.421462    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8522a2fc-372a-4849-9c0e-a9033437657f-cabundle-cert\") pod \"8522a2fc-372a-4849-9c0e-a9033437657f\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") "
Sep 17 18:20:45.421768 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.421494    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d452\" (UniqueName: \"kubernetes.io/projected/8522a2fc-372a-4849-9c0e-a9033437657f-kube-api-access-7d452\") pod \"8522a2fc-372a-4849-9c0e-a9033437657f\" (UID: \"8522a2fc-372a-4849-9c0e-a9033437657f\") "
Sep 17 18:20:45.421768 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.421738    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8522a2fc-372a-4849-9c0e-a9033437657f" (UID: "8522a2fc-372a-4849-9c0e-a9033437657f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:45.421927 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.421784    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-home" (OuterVolumeSpecName: "home") pod "8522a2fc-372a-4849-9c0e-a9033437657f" (UID: "8522a2fc-372a-4849-9c0e-a9033437657f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:45.422029 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.422002    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8522a2fc-372a-4849-9c0e-a9033437657f" (UID: "8522a2fc-372a-4849-9c0e-a9033437657f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:45.422090 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.422051    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8522a2fc-372a-4849-9c0e-a9033437657f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8522a2fc-372a-4849-9c0e-a9033437657f" (UID: "8522a2fc-372a-4849-9c0e-a9033437657f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:45.424257 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.424227    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8522a2fc-372a-4849-9c0e-a9033437657f-kube-api-access-7d452" (OuterVolumeSpecName: "kube-api-access-7d452") pod "8522a2fc-372a-4849-9c0e-a9033437657f" (UID: "8522a2fc-372a-4849-9c0e-a9033437657f"). InnerVolumeSpecName "kube-api-access-7d452". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:45.426756 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.426720    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8522a2fc-372a-4849-9c0e-a9033437657f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8522a2fc-372a-4849-9c0e-a9033437657f" (UID: "8522a2fc-372a-4849-9c0e-a9033437657f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:45.426873 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.426822    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-dshm" (OuterVolumeSpecName: "dshm") pod "8522a2fc-372a-4849-9c0e-a9033437657f" (UID: "8522a2fc-372a-4849-9c0e-a9033437657f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:45.438717 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.438629    2574 generic.go:356] "Generic (PLEG): container finished" podID="8522a2fc-372a-4849-9c0e-a9033437657f" containerID="63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f" exitCode=0
Sep 17 18:20:45.438717 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.438695    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"
Sep 17 18:20:45.438914 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.438695    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld" event={"ID":"8522a2fc-372a-4849-9c0e-a9033437657f","Type":"ContainerDied","Data":"63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f"}
Sep 17 18:20:45.438914 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.438844    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld" event={"ID":"8522a2fc-372a-4849-9c0e-a9033437657f","Type":"ContainerDied","Data":"124495bb382a3e8c070997c1f8a161f2f3189a6c90e4a78aa7495d7d4d2d37c2"}
Sep 17 18:20:45.438914 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.438868    2574 scope.go:117] "RemoveContainer" containerID="63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f"
Sep 17 18:20:45.445212 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.445183    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8522a2fc-372a-4849-9c0e-a9033437657f" (UID: "8522a2fc-372a-4849-9c0e-a9033437657f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:45.477494 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.477471    2574 scope.go:117] "RemoveContainer" containerID="8224e2ab119fc926e25a9d8a8f3b3a531b64f64ba510ef969a7c7ff3232b5708"
Sep 17 18:20:45.486238 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.486218    2574 scope.go:117] "RemoveContainer" containerID="63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f"
Sep 17 18:20:45.486560 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:20:45.486541    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f\": container with ID starting with 63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f not found: ID does not exist" containerID="63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f"
Sep 17 18:20:45.486623 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.486570    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f"} err="failed to get container status \"63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f\": rpc error: code = NotFound desc = could not find container \"63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f\": container with ID starting with 63e7c29a99dcbacfadd3c11315c2af683ee02cb8d3811c52e6ef1847e186122f not found: ID does not exist"
Sep 17 18:20:45.486623 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.486590    2574 scope.go:117] "RemoveContainer" containerID="8224e2ab119fc926e25a9d8a8f3b3a531b64f64ba510ef969a7c7ff3232b5708"
Sep 17 18:20:45.486843 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:20:45.486827    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8224e2ab119fc926e25a9d8a8f3b3a531b64f64ba510ef969a7c7ff3232b5708\": container with ID starting with 8224e2ab119fc926e25a9d8a8f3b3a531b64f64ba510ef969a7c7ff3232b5708 not found: ID does not exist" containerID="8224e2ab119fc926e25a9d8a8f3b3a531b64f64ba510ef969a7c7ff3232b5708"
Sep 17 18:20:45.486881 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.486848    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8224e2ab119fc926e25a9d8a8f3b3a531b64f64ba510ef969a7c7ff3232b5708"} err="failed to get container status \"8224e2ab119fc926e25a9d8a8f3b3a531b64f64ba510ef969a7c7ff3232b5708\": rpc error: code = NotFound desc = could not find container \"8224e2ab119fc926e25a9d8a8f3b3a531b64f64ba510ef969a7c7ff3232b5708\": container with ID starting with 8224e2ab119fc926e25a9d8a8f3b3a531b64f64ba510ef969a7c7ff3232b5708 not found: ID does not exist"
Sep 17 18:20:45.522564 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.522537    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8522a2fc-372a-4849-9c0e-a9033437657f-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:20:45.522564 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.522565    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d452\" (UniqueName: \"kubernetes.io/projected/8522a2fc-372a-4849-9c0e-a9033437657f-kube-api-access-7d452\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:20:45.522732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.522580    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:20:45.522732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.522593    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8522a2fc-372a-4849-9c0e-a9033437657f-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:20:45.522732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.522606    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:20:45.522732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.522639    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:20:45.522732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.522652    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:20:45.522732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.522663    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8522a2fc-372a-4849-9c0e-a9033437657f-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:20:45.650330 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.650302    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:20:45.724846 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.724748    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14e573ca-198d-429c-8a24-4401b9b7c00b-cabundle-cert\") pod \"14e573ca-198d-429c-8a24-4401b9b7c00b\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") "
Sep 17 18:20:45.724846 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.724799    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14e573ca-198d-429c-8a24-4401b9b7c00b-kserve-provision-location\") pod \"14e573ca-198d-429c-8a24-4401b9b7c00b\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") "
Sep 17 18:20:45.725085 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.724898    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e573ca-198d-429c-8a24-4401b9b7c00b-tmp-dir\") pod \"14e573ca-198d-429c-8a24-4401b9b7c00b\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") "
Sep 17 18:20:45.725085 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.724942    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e573ca-198d-429c-8a24-4401b9b7c00b-tls-certs\") pod \"14e573ca-198d-429c-8a24-4401b9b7c00b\" (UID: \"14e573ca-198d-429c-8a24-4401b9b7c00b\") "
Sep 17 18:20:45.725189 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.725134    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e573ca-198d-429c-8a24-4401b9b7c00b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "14e573ca-198d-429c-8a24-4401b9b7c00b" (UID: "14e573ca-198d-429c-8a24-4401b9b7c00b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:45.725308 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.725284    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e573ca-198d-429c-8a24-4401b9b7c00b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14e573ca-198d-429c-8a24-4401b9b7c00b" (UID: "14e573ca-198d-429c-8a24-4401b9b7c00b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:45.727268 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.727247    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e573ca-198d-429c-8a24-4401b9b7c00b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14e573ca-198d-429c-8a24-4401b9b7c00b" (UID: "14e573ca-198d-429c-8a24-4401b9b7c00b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:45.746488 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.746411    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e573ca-198d-429c-8a24-4401b9b7c00b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "14e573ca-198d-429c-8a24-4401b9b7c00b" (UID: "14e573ca-198d-429c-8a24-4401b9b7c00b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:45.762069 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.762042    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"]
Sep 17 18:20:45.765362 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.765335    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-xkqld"]
Sep 17 18:20:45.826343 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.826311    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e573ca-198d-429c-8a24-4401b9b7c00b-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:20:45.826343 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.826338    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e573ca-198d-429c-8a24-4401b9b7c00b-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:20:45.826343 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.826346    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14e573ca-198d-429c-8a24-4401b9b7c00b-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:20:45.826621 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.826356    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14e573ca-198d-429c-8a24-4401b9b7c00b-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:20:45.938145 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:45.938102    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8522a2fc-372a-4849-9c0e-a9033437657f" path="/var/lib/kubelet/pods/8522a2fc-372a-4849-9c0e-a9033437657f/volumes"
Sep 17 18:20:46.446552 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:46.446449    2574 generic.go:356] "Generic (PLEG): container finished" podID="14e573ca-198d-429c-8a24-4401b9b7c00b" containerID="5ab7a164e81ded33443d0fd63087db3d1ffe721b5fc3b0dea3f27366f9ffb069" exitCode=0
Sep 17 18:20:46.446552 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:46.446535    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"
Sep 17 18:20:46.446775 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:46.446551    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn" event={"ID":"14e573ca-198d-429c-8a24-4401b9b7c00b","Type":"ContainerDied","Data":"5ab7a164e81ded33443d0fd63087db3d1ffe721b5fc3b0dea3f27366f9ffb069"}
Sep 17 18:20:46.446775 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:46.446611    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn" event={"ID":"14e573ca-198d-429c-8a24-4401b9b7c00b","Type":"ContainerDied","Data":"4dacff86ccf55b09326079ed00ae694ec651dbde3957494c0621ab7ee44d656e"}
Sep 17 18:20:46.446775 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:46.446640    2574 scope.go:117] "RemoveContainer" containerID="5ab7a164e81ded33443d0fd63087db3d1ffe721b5fc3b0dea3f27366f9ffb069"
Sep 17 18:20:46.455926 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:46.455909    2574 scope.go:117] "RemoveContainer" containerID="29637d1551063601653e88c909d4fdd32fc2389e2e34919189006602ea71bbf7"
Sep 17 18:20:46.463846 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:46.463819    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"]
Sep 17 18:20:46.465372 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:46.465355    2574 scope.go:117] "RemoveContainer" containerID="5ab7a164e81ded33443d0fd63087db3d1ffe721b5fc3b0dea3f27366f9ffb069"
Sep 17 18:20:46.465681 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:20:46.465660    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab7a164e81ded33443d0fd63087db3d1ffe721b5fc3b0dea3f27366f9ffb069\": container with ID starting with 5ab7a164e81ded33443d0fd63087db3d1ffe721b5fc3b0dea3f27366f9ffb069 not found: ID does not exist" containerID="5ab7a164e81ded33443d0fd63087db3d1ffe721b5fc3b0dea3f27366f9ffb069"
Sep 17 18:20:46.465735 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:46.465689    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab7a164e81ded33443d0fd63087db3d1ffe721b5fc3b0dea3f27366f9ffb069"} err="failed to get container status \"5ab7a164e81ded33443d0fd63087db3d1ffe721b5fc3b0dea3f27366f9ffb069\": rpc error: code = NotFound desc = could not find container \"5ab7a164e81ded33443d0fd63087db3d1ffe721b5fc3b0dea3f27366f9ffb069\": container with ID starting with 5ab7a164e81ded33443d0fd63087db3d1ffe721b5fc3b0dea3f27366f9ffb069 not found: ID does not exist"
Sep 17 18:20:46.465735 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:46.465707    2574 scope.go:117] "RemoveContainer" containerID="29637d1551063601653e88c909d4fdd32fc2389e2e34919189006602ea71bbf7"
Sep 17 18:20:46.465948 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:20:46.465928    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29637d1551063601653e88c909d4fdd32fc2389e2e34919189006602ea71bbf7\": container with ID starting with 29637d1551063601653e88c909d4fdd32fc2389e2e34919189006602ea71bbf7 not found: ID does not exist" containerID="29637d1551063601653e88c909d4fdd32fc2389e2e34919189006602ea71bbf7"
Sep 17 18:20:46.465986 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:46.465953    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29637d1551063601653e88c909d4fdd32fc2389e2e34919189006602ea71bbf7"} err="failed to get container status \"29637d1551063601653e88c909d4fdd32fc2389e2e34919189006602ea71bbf7\": rpc error: code = NotFound desc = could not find container \"29637d1551063601653e88c909d4fdd32fc2389e2e34919189006602ea71bbf7\": container with ID starting with 29637d1551063601653e88c909d4fdd32fc2389e2e34919189006602ea71bbf7 not found: ID does not exist"
Sep 17 18:20:46.467059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:46.467039    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6999f94d6b-jsckn"]
Sep 17 18:20:47.938642 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:20:47.938607    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14e573ca-198d-429c-8a24-4401b9b7c00b" path="/var/lib/kubelet/pods/14e573ca-198d-429c-8a24-4401b9b7c00b/volumes"
Sep 17 18:21:10.718148 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:10.718069    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:10.777075 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:10.777042    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:34.513308 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:34.513270    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:34.612458 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:34.612419    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:40.896295 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:40.896258    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:43.980316 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.980273    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"]
Sep 17 18:21:43.980938 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.980917    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8522a2fc-372a-4849-9c0e-a9033437657f" containerName="main"
Sep 17 18:21:43.981000 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.980942    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8522a2fc-372a-4849-9c0e-a9033437657f" containerName="main"
Sep 17 18:21:43.981000 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.980952    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14e573ca-198d-429c-8a24-4401b9b7c00b" containerName="main"
Sep 17 18:21:43.981000 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.980957    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="14e573ca-198d-429c-8a24-4401b9b7c00b" containerName="main"
Sep 17 18:21:43.981000 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.980972    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8522a2fc-372a-4849-9c0e-a9033437657f" containerName="storage-initializer"
Sep 17 18:21:43.981000 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.980979    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8522a2fc-372a-4849-9c0e-a9033437657f" containerName="storage-initializer"
Sep 17 18:21:43.981000 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.980992    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14e573ca-198d-429c-8a24-4401b9b7c00b" containerName="storage-initializer"
Sep 17 18:21:43.981000 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.980998    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="14e573ca-198d-429c-8a24-4401b9b7c00b" containerName="storage-initializer"
Sep 17 18:21:43.981323 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.981053    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8522a2fc-372a-4849-9c0e-a9033437657f" containerName="main"
Sep 17 18:21:43.981323 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.981061    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="14e573ca-198d-429c-8a24-4401b9b7c00b" containerName="main"
Sep 17 18:21:43.983577 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.983558    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:43.985981 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.985958    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 18:21:43.986733 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.986707    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:21:43.986829 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.986776    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 18:21:43.986829 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.986707    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 18:21:43.986829 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.986711    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-6tcbp\""
Sep 17 18:21:43.987046 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.987029    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 18:21:43.996925 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:43.996896    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"]
Sep 17 18:21:44.013750 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.013717    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:44.019849 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.019824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a156ff55-8694-4ef6-bb68-b940856f665f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.020018 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.019998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-home\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.020065 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.020040    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.020113 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.020090    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a156ff55-8694-4ef6-bb68-b940856f665f-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.020212 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.020194    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgvlx\" (UniqueName: \"kubernetes.io/projected/a156ff55-8694-4ef6-bb68-b940856f665f-kube-api-access-cgvlx\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.020246 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.020235    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.020289 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.020278    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-dshm\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.020325 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.020306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.121441 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.121399    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-home\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.121441 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.121441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.121728 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.121459    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a156ff55-8694-4ef6-bb68-b940856f665f-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.121728 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.121493    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgvlx\" (UniqueName: \"kubernetes.io/projected/a156ff55-8694-4ef6-bb68-b940856f665f-kube-api-access-cgvlx\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.121728 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.121629    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.121728 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.121685    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-dshm\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.121728 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.121719    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.121997 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.121756    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a156ff55-8694-4ef6-bb68-b940856f665f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.121997 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.121902    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.121997 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.121940    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-home\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.122161 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.122023    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.122161 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.122086    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.122397 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.122375    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a156ff55-8694-4ef6-bb68-b940856f665f-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.124102 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.124076    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-dshm\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.124372 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.124356    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a156ff55-8694-4ef6-bb68-b940856f665f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.129902 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.129879    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgvlx\" (UniqueName: \"kubernetes.io/projected/a156ff55-8694-4ef6-bb68-b940856f665f-kube-api-access-cgvlx\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-hchmd\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.208976 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.208940    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"]
Sep 17 18:21:44.213011 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.212981    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:44.225706 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.225661    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"]
Sep 17 18:21:44.249540 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.249421    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:44.295540 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.295478    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:44.323798 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.323771    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6794df0-d182-4b97-ba03-f7a372c51dd1-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:44.323958 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.323812    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6794df0-d182-4b97-ba03-f7a372c51dd1-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:44.323958 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.323850    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6794df0-d182-4b97-ba03-f7a372c51dd1-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:44.323958 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.323875    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6794df0-d182-4b97-ba03-f7a372c51dd1-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:44.339756 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.339722    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:44.425179 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.425134    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6794df0-d182-4b97-ba03-f7a372c51dd1-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:44.425388 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.425205    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6794df0-d182-4b97-ba03-f7a372c51dd1-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:44.425388 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.425239    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6794df0-d182-4b97-ba03-f7a372c51dd1-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:44.425388 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.425329    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6794df0-d182-4b97-ba03-f7a372c51dd1-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:44.425596 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.425505    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6794df0-d182-4b97-ba03-f7a372c51dd1-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:44.425596 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.425568    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6794df0-d182-4b97-ba03-f7a372c51dd1-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:44.426004 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.425981    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6794df0-d182-4b97-ba03-f7a372c51dd1-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:44.428042 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.428014    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6794df0-d182-4b97-ba03-f7a372c51dd1-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:44.527848 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.527816    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:44.574325 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.574292    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"]
Sep 17 18:21:44.575103 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:21:44.575069    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda156ff55_8694_4ef6_bb68_b940856f665f.slice/crio-a61f7202b76a4087e28e836edf8c1529689033f2363863d665c42b19b82d2055 WatchSource:0}: Error finding container a61f7202b76a4087e28e836edf8c1529689033f2363863d665c42b19b82d2055: Status 404 returned error can't find the container with id a61f7202b76a4087e28e836edf8c1529689033f2363863d665c42b19b82d2055
Sep 17 18:21:44.769211 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.769167    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd" event={"ID":"a156ff55-8694-4ef6-bb68-b940856f665f","Type":"ContainerStarted","Data":"6195edd3d32365a25f8e579b90e78fbe07047d03631ca63b41ffa6a1cd3b44eb"}
Sep 17 18:21:44.769414 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.769254    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd" event={"ID":"a156ff55-8694-4ef6-bb68-b940856f665f","Type":"ContainerStarted","Data":"a61f7202b76a4087e28e836edf8c1529689033f2363863d665c42b19b82d2055"}
Sep 17 18:21:44.804378 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:44.804351    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"]
Sep 17 18:21:44.806620 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:21:44.806593    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6794df0_d182_4b97_ba03_f7a372c51dd1.slice/crio-d1167276029137398e024470e92d731cd885ddb01645ca5cf8117f949d512940 WatchSource:0}: Error finding container d1167276029137398e024470e92d731cd885ddb01645ca5cf8117f949d512940: Status 404 returned error can't find the container with id d1167276029137398e024470e92d731cd885ddb01645ca5cf8117f949d512940
Sep 17 18:21:45.776803 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:45.776765    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m" event={"ID":"c6794df0-d182-4b97-ba03-f7a372c51dd1","Type":"ContainerStarted","Data":"8bfc85a9fa855c35688734395264abe078d0c485667b61bf0235e5eeb000d3f1"}
Sep 17 18:21:45.777296 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:45.776859    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m" event={"ID":"c6794df0-d182-4b97-ba03-f7a372c51dd1","Type":"ContainerStarted","Data":"d1167276029137398e024470e92d731cd885ddb01645ca5cf8117f949d512940"}
Sep 17 18:21:47.058863 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:47.058824    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:47.161284 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:47.161216    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:47.788792 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:47.788758    2574 generic.go:356] "Generic (PLEG): container finished" podID="c6794df0-d182-4b97-ba03-f7a372c51dd1" containerID="8bfc85a9fa855c35688734395264abe078d0c485667b61bf0235e5eeb000d3f1" exitCode=0
Sep 17 18:21:47.788998 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:47.788841    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m" event={"ID":"c6794df0-d182-4b97-ba03-f7a372c51dd1","Type":"ContainerDied","Data":"8bfc85a9fa855c35688734395264abe078d0c485667b61bf0235e5eeb000d3f1"}
Sep 17 18:21:47.790711 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:47.790682    2574 generic.go:356] "Generic (PLEG): container finished" podID="a156ff55-8694-4ef6-bb68-b940856f665f" containerID="6195edd3d32365a25f8e579b90e78fbe07047d03631ca63b41ffa6a1cd3b44eb" exitCode=0
Sep 17 18:21:47.790817 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:47.790754    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd" event={"ID":"a156ff55-8694-4ef6-bb68-b940856f665f","Type":"ContainerDied","Data":"6195edd3d32365a25f8e579b90e78fbe07047d03631ca63b41ffa6a1cd3b44eb"}
Sep 17 18:21:48.799200 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:48.799152    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m" event={"ID":"c6794df0-d182-4b97-ba03-f7a372c51dd1","Type":"ContainerStarted","Data":"38d44472b80af2817e925cc36ce7533d49fd8a68b0d531b8f795b9c6adea47e9"}
Sep 17 18:21:48.799696 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:48.799403    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:21:48.801370 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:48.801344    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd" event={"ID":"a156ff55-8694-4ef6-bb68-b940856f665f","Type":"ContainerStarted","Data":"5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49"}
Sep 17 18:21:48.819326 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:48.819277    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m" podStartSLOduration=2.274840693 podStartE2EDuration="4.819261677s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.544420984s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:48.817665065 +0000 UTC m=+1479.118487406" watchObservedRunningTime="2026-09-17 18:21:48.819261677 +0000 UTC m=+1479.120084033"
Sep 17 18:21:48.844268 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:48.844212    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd" podStartSLOduration=3.305158868 podStartE2EDuration="5.844194698s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.53903583s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:48.842032021 +0000 UTC m=+1479.142854382" watchObservedRunningTime="2026-09-17 18:21:48.844194698 +0000 UTC m=+1479.145017021"
Sep 17 18:21:49.296310 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:49.296266    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:54.296507 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:54.296467    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:54.309504 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:54.309474    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:54.850056 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:54.850017    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:21:58.754352 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:21:58.754312    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:11.895065 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:11.895033    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:22:11.902594 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:11.902572    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:22:19.819220 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:19.819181    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:22:20.912032 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:20.911954    2574 scope.go:117] "RemoveContainer" containerID="ff48ea8be22dffc0110a65d0cd1ad2757251abb2b4224e125b62a427dce03067"
Sep 17 18:22:21.221831 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.221749    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m_c6794df0-d182-4b97-ba03-f7a372c51dd1/main/0.log"
Sep 17 18:22:21.408529 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.408481    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"]
Sep 17 18:22:21.408718 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.408533    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:21.413267 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.413239    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"]
Sep 17 18:22:21.413506 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.413485    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m" podUID="c6794df0-d182-4b97-ba03-f7a372c51dd1" containerName="main" containerID="cri-o://38d44472b80af2817e925cc36ce7533d49fd8a68b0d531b8f795b9c6adea47e9" gracePeriod=30
Sep 17 18:22:21.420732 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:22:21.420697    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:22:21.420732 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:22:21.420732 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:22:21.420732 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49" cmd=["/bin/sleep","15"]
Sep 17 18:22:21.421033 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.420751    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd" message="" err=<
Sep 17 18:22:21.421033 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:22:21.421033 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:22:21.421033 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 18:22:21.421033 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:22:21.420763    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:22:21.421033 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:22:21.421033 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:22:21.421033 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd" podUID="a156ff55-8694-4ef6-bb68-b940856f665f" containerName="main" containerID="cri-o://5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49"
Sep 17 18:22:21.421033 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.420809    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd" podUID="a156ff55-8694-4ef6-bb68-b940856f665f" containerName="main" containerID="cri-o://5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49" gracePeriod=60
Sep 17 18:22:21.484110 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.484003    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:21.797827 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.797798    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:22:21.854438 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.854404    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgvlx\" (UniqueName: \"kubernetes.io/projected/a156ff55-8694-4ef6-bb68-b940856f665f-kube-api-access-cgvlx\") pod \"a156ff55-8694-4ef6-bb68-b940856f665f\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") "
Sep 17 18:22:21.854665 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.854455    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-tmp-dir\") pod \"a156ff55-8694-4ef6-bb68-b940856f665f\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") "
Sep 17 18:22:21.854665 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.854493    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-home\") pod \"a156ff55-8694-4ef6-bb68-b940856f665f\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") "
Sep 17 18:22:21.854665 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.854532    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-kserve-provision-location\") pod \"a156ff55-8694-4ef6-bb68-b940856f665f\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") "
Sep 17 18:22:21.854665 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.854570    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a156ff55-8694-4ef6-bb68-b940856f665f-tls-certs\") pod \"a156ff55-8694-4ef6-bb68-b940856f665f\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") "
Sep 17 18:22:21.854665 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.854592    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-dshm\") pod \"a156ff55-8694-4ef6-bb68-b940856f665f\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") "
Sep 17 18:22:21.854952 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.854717    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-model-cache\") pod \"a156ff55-8694-4ef6-bb68-b940856f665f\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") "
Sep 17 18:22:21.854952 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.854778    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a156ff55-8694-4ef6-bb68-b940856f665f-cabundle-cert\") pod \"a156ff55-8694-4ef6-bb68-b940856f665f\" (UID: \"a156ff55-8694-4ef6-bb68-b940856f665f\") "
Sep 17 18:22:21.855080 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.855045    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a156ff55-8694-4ef6-bb68-b940856f665f" (UID: "a156ff55-8694-4ef6-bb68-b940856f665f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:21.855187 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.855158    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a156ff55-8694-4ef6-bb68-b940856f665f" (UID: "a156ff55-8694-4ef6-bb68-b940856f665f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:21.855349 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.855292    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a156ff55-8694-4ef6-bb68-b940856f665f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a156ff55-8694-4ef6-bb68-b940856f665f" (UID: "a156ff55-8694-4ef6-bb68-b940856f665f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:21.855668 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.855647    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-home" (OuterVolumeSpecName: "home") pod "a156ff55-8694-4ef6-bb68-b940856f665f" (UID: "a156ff55-8694-4ef6-bb68-b940856f665f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:21.857161 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.857138    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-dshm" (OuterVolumeSpecName: "dshm") pod "a156ff55-8694-4ef6-bb68-b940856f665f" (UID: "a156ff55-8694-4ef6-bb68-b940856f665f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:21.857548 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.857501    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a156ff55-8694-4ef6-bb68-b940856f665f-kube-api-access-cgvlx" (OuterVolumeSpecName: "kube-api-access-cgvlx") pod "a156ff55-8694-4ef6-bb68-b940856f665f" (UID: "a156ff55-8694-4ef6-bb68-b940856f665f"). InnerVolumeSpecName "kube-api-access-cgvlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:21.857647 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.857581    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a156ff55-8694-4ef6-bb68-b940856f665f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a156ff55-8694-4ef6-bb68-b940856f665f" (UID: "a156ff55-8694-4ef6-bb68-b940856f665f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:21.890675 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.890592    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a156ff55-8694-4ef6-bb68-b940856f665f" (UID: "a156ff55-8694-4ef6-bb68-b940856f665f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:21.955726 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.955698    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.955726 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.955726    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.956118 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.955736    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a156ff55-8694-4ef6-bb68-b940856f665f-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.956118 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.955745    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.956118 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.955753    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.956118 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.955763    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a156ff55-8694-4ef6-bb68-b940856f665f-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.956118 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.955771    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgvlx\" (UniqueName: \"kubernetes.io/projected/a156ff55-8694-4ef6-bb68-b940856f665f-kube-api-access-cgvlx\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.956118 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.955799    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a156ff55-8694-4ef6-bb68-b940856f665f-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.995397 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.995357    2574 generic.go:356] "Generic (PLEG): container finished" podID="c6794df0-d182-4b97-ba03-f7a372c51dd1" containerID="38d44472b80af2817e925cc36ce7533d49fd8a68b0d531b8f795b9c6adea47e9" exitCode=0
Sep 17 18:22:21.995616 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.995436    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m" event={"ID":"c6794df0-d182-4b97-ba03-f7a372c51dd1","Type":"ContainerDied","Data":"38d44472b80af2817e925cc36ce7533d49fd8a68b0d531b8f795b9c6adea47e9"}
Sep 17 18:22:21.997315 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.997284    2574 generic.go:356] "Generic (PLEG): container finished" podID="a156ff55-8694-4ef6-bb68-b940856f665f" containerID="5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49" exitCode=0
Sep 17 18:22:21.997457 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.997365    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd" event={"ID":"a156ff55-8694-4ef6-bb68-b940856f665f","Type":"ContainerDied","Data":"5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49"}
Sep 17 18:22:21.997457 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.997375    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"
Sep 17 18:22:21.997457 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.997415    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd" event={"ID":"a156ff55-8694-4ef6-bb68-b940856f665f","Type":"ContainerDied","Data":"a61f7202b76a4087e28e836edf8c1529689033f2363863d665c42b19b82d2055"}
Sep 17 18:22:21.997457 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:21.997445    2574 scope.go:117] "RemoveContainer" containerID="5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49"
Sep 17 18:22:22.010673 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.010648    2574 scope.go:117] "RemoveContainer" containerID="6195edd3d32365a25f8e579b90e78fbe07047d03631ca63b41ffa6a1cd3b44eb"
Sep 17 18:22:22.021669 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.021574    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"]
Sep 17 18:22:22.023737 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.023717    2574 scope.go:117] "RemoveContainer" containerID="5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49"
Sep 17 18:22:22.024002 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.023976    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-hchmd"]
Sep 17 18:22:22.024153 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:22:22.024134    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49\": container with ID starting with 5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49 not found: ID does not exist" containerID="5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49"
Sep 17 18:22:22.024199 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.024166    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49"} err="failed to get container status \"5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49\": rpc error: code = NotFound desc = could not find container \"5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49\": container with ID starting with 5e2e2ac77cf32c053e8842fcb4f8f1b4c5bbb08458c0547281e08fbc24734f49 not found: ID does not exist"
Sep 17 18:22:22.024199 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.024186    2574 scope.go:117] "RemoveContainer" containerID="6195edd3d32365a25f8e579b90e78fbe07047d03631ca63b41ffa6a1cd3b44eb"
Sep 17 18:22:22.024548 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:22:22.024497    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6195edd3d32365a25f8e579b90e78fbe07047d03631ca63b41ffa6a1cd3b44eb\": container with ID starting with 6195edd3d32365a25f8e579b90e78fbe07047d03631ca63b41ffa6a1cd3b44eb not found: ID does not exist" containerID="6195edd3d32365a25f8e579b90e78fbe07047d03631ca63b41ffa6a1cd3b44eb"
Sep 17 18:22:22.024598 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.024558    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6195edd3d32365a25f8e579b90e78fbe07047d03631ca63b41ffa6a1cd3b44eb"} err="failed to get container status \"6195edd3d32365a25f8e579b90e78fbe07047d03631ca63b41ffa6a1cd3b44eb\": rpc error: code = NotFound desc = could not find container \"6195edd3d32365a25f8e579b90e78fbe07047d03631ca63b41ffa6a1cd3b44eb\": container with ID starting with 6195edd3d32365a25f8e579b90e78fbe07047d03631ca63b41ffa6a1cd3b44eb not found: ID does not exist"
Sep 17 18:22:22.078249 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.078222    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:22:22.158011 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.157954    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6794df0-d182-4b97-ba03-f7a372c51dd1-tls-certs\") pod \"c6794df0-d182-4b97-ba03-f7a372c51dd1\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") "
Sep 17 18:22:22.158232 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.158046    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6794df0-d182-4b97-ba03-f7a372c51dd1-tmp-dir\") pod \"c6794df0-d182-4b97-ba03-f7a372c51dd1\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") "
Sep 17 18:22:22.158232 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.158094    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6794df0-d182-4b97-ba03-f7a372c51dd1-kserve-provision-location\") pod \"c6794df0-d182-4b97-ba03-f7a372c51dd1\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") "
Sep 17 18:22:22.158232 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.158145    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6794df0-d182-4b97-ba03-f7a372c51dd1-cabundle-cert\") pod \"c6794df0-d182-4b97-ba03-f7a372c51dd1\" (UID: \"c6794df0-d182-4b97-ba03-f7a372c51dd1\") "
Sep 17 18:22:22.158548 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.158492    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6794df0-d182-4b97-ba03-f7a372c51dd1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c6794df0-d182-4b97-ba03-f7a372c51dd1" (UID: "c6794df0-d182-4b97-ba03-f7a372c51dd1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:22.158548 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.158535    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6794df0-d182-4b97-ba03-f7a372c51dd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6794df0-d182-4b97-ba03-f7a372c51dd1" (UID: "c6794df0-d182-4b97-ba03-f7a372c51dd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:22.160554 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.160502    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6794df0-d182-4b97-ba03-f7a372c51dd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6794df0-d182-4b97-ba03-f7a372c51dd1" (UID: "c6794df0-d182-4b97-ba03-f7a372c51dd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:22.182494 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.182450    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6794df0-d182-4b97-ba03-f7a372c51dd1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6794df0-d182-4b97-ba03-f7a372c51dd1" (UID: "c6794df0-d182-4b97-ba03-f7a372c51dd1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:22.259765 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.259716    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6794df0-d182-4b97-ba03-f7a372c51dd1-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:22:22.259765 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.259758    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6794df0-d182-4b97-ba03-f7a372c51dd1-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:22:22.259765 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.259767    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6794df0-d182-4b97-ba03-f7a372c51dd1-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:22:22.259765 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:22.259775    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6794df0-d182-4b97-ba03-f7a372c51dd1-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:22:23.004924 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:23.004881    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"
Sep 17 18:22:23.005406 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:23.004886    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m" event={"ID":"c6794df0-d182-4b97-ba03-f7a372c51dd1","Type":"ContainerDied","Data":"d1167276029137398e024470e92d731cd885ddb01645ca5cf8117f949d512940"}
Sep 17 18:22:23.005406 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:23.005062    2574 scope.go:117] "RemoveContainer" containerID="38d44472b80af2817e925cc36ce7533d49fd8a68b0d531b8f795b9c6adea47e9"
Sep 17 18:22:23.014586 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:23.014562    2574 scope.go:117] "RemoveContainer" containerID="8bfc85a9fa855c35688734395264abe078d0c485667b61bf0235e5eeb000d3f1"
Sep 17 18:22:23.026926 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:23.026899    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"]
Sep 17 18:22:23.032967 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:23.032937    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-69bd888664-d4r6m"]
Sep 17 18:22:23.938646 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:23.938611    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a156ff55-8694-4ef6-bb68-b940856f665f" path="/var/lib/kubelet/pods/a156ff55-8694-4ef6-bb68-b940856f665f/volumes"
Sep 17 18:22:23.939056 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:23.939043    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6794df0-d182-4b97-ba03-f7a372c51dd1" path="/var/lib/kubelet/pods/c6794df0-d182-4b97-ba03-f7a372c51dd1/volumes"
Sep 17 18:22:39.409850 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:39.409776    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:39.509850 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:39.509816    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:57.679574 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:57.679537    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:57.738560 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:22:57.738506    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:11.618748 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:11.618679    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:11.712445 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:11.712412    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:17.914503 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:17.914471    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:20.930563 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:20.930507    2574 scope.go:117] "RemoveContainer" containerID="23fbaf70f569eacaea0beb81f31c960c8cd4b618af3d2a2e0cdd931bbabfbe30"
Sep 17 18:23:21.466943 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.466899    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"]
Sep 17 18:23:21.467431 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.467405    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6794df0-d182-4b97-ba03-f7a372c51dd1" containerName="storage-initializer"
Sep 17 18:23:21.467431 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.467434    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6794df0-d182-4b97-ba03-f7a372c51dd1" containerName="storage-initializer"
Sep 17 18:23:21.467707 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.467467    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a156ff55-8694-4ef6-bb68-b940856f665f" containerName="storage-initializer"
Sep 17 18:23:21.467707 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.467477    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a156ff55-8694-4ef6-bb68-b940856f665f" containerName="storage-initializer"
Sep 17 18:23:21.467707 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.467496    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6794df0-d182-4b97-ba03-f7a372c51dd1" containerName="main"
Sep 17 18:23:21.467707 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.467502    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6794df0-d182-4b97-ba03-f7a372c51dd1" containerName="main"
Sep 17 18:23:21.467707 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.467539    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a156ff55-8694-4ef6-bb68-b940856f665f" containerName="main"
Sep 17 18:23:21.467707 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.467547    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a156ff55-8694-4ef6-bb68-b940856f665f" containerName="main"
Sep 17 18:23:21.467707 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.467644    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a156ff55-8694-4ef6-bb68-b940856f665f" containerName="main"
Sep 17 18:23:21.467707 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.467655    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6794df0-d182-4b97-ba03-f7a372c51dd1" containerName="main"
Sep 17 18:23:21.470655 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.470638    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.473106 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.473082    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 18:23:21.473238 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.473191    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 18:23:21.474025 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.474004    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 18:23:21.474141 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.474074    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 18:23:21.474141 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.474135    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:23:21.474272 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.474140    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-9gjr9\""
Sep 17 18:23:21.480685 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.480663    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"]
Sep 17 18:23:21.490755 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.490685    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:21.569102 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.569067    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5289e7a3-21ce-4848-beb1-ba5b0e212f81-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.569102 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.569104    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5289e7a3-21ce-4848-beb1-ba5b0e212f81-tls-certs\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.569327 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.569130    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tqzf\" (UniqueName: \"kubernetes.io/projected/5289e7a3-21ce-4848-beb1-ba5b0e212f81-kube-api-access-9tqzf\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.569327 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.569197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-model-cache\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.569327 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.569260    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-tmp-dir\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.569327 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.569283    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-home\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.569327 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.569304    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-dshm\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.569492 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.569330    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.670308 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.670268    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5289e7a3-21ce-4848-beb1-ba5b0e212f81-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.670540 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.670326    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5289e7a3-21ce-4848-beb1-ba5b0e212f81-tls-certs\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.670540 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.670357    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tqzf\" (UniqueName: \"kubernetes.io/projected/5289e7a3-21ce-4848-beb1-ba5b0e212f81-kube-api-access-9tqzf\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.670540 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.670386    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-model-cache\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.670540 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.670431    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-tmp-dir\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.670795 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.670579    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-home\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.670795 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.670630    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-dshm\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.670795 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.670682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.670955 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.670820    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-model-cache\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.671019 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.670959    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-home\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.671075 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.671022    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-tmp-dir\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.671165 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.671124    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.671165 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.671143    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5289e7a3-21ce-4848-beb1-ba5b0e212f81-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.673096 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.673070    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-dshm\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.673247 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.673230    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5289e7a3-21ce-4848-beb1-ba5b0e212f81-tls-certs\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.681246 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.681211    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tqzf\" (UniqueName: \"kubernetes.io/projected/5289e7a3-21ce-4848-beb1-ba5b0e212f81-kube-api-access-9tqzf\") pod \"tokenizer-template-args-test-kserve-698c9576bb-4s5gh\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.721874 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.721781    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"]
Sep 17 18:23:21.725275 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.725235    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:21.739377 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.739347    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"]
Sep 17 18:23:21.766045 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.766011    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:21.771371 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.771324    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/464b7491-c6a1-4927-95f2-f0bd4ff796ae-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:21.771646 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.771425    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464b7491-c6a1-4927-95f2-f0bd4ff796ae-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:21.771646 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.771472    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/464b7491-c6a1-4927-95f2-f0bd4ff796ae-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:21.771646 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.771612    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/464b7491-c6a1-4927-95f2-f0bd4ff796ae-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:21.782398 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.782371    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:21.835146 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.835058    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:21.872065 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.872025    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/464b7491-c6a1-4927-95f2-f0bd4ff796ae-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:21.872285 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.872122    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464b7491-c6a1-4927-95f2-f0bd4ff796ae-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:21.872285 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.872150    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/464b7491-c6a1-4927-95f2-f0bd4ff796ae-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:21.872285 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.872185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/464b7491-c6a1-4927-95f2-f0bd4ff796ae-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:21.872623 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.872601    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464b7491-c6a1-4927-95f2-f0bd4ff796ae-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:21.872711 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.872637    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/464b7491-c6a1-4927-95f2-f0bd4ff796ae-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:21.872969 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.872946    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/464b7491-c6a1-4927-95f2-f0bd4ff796ae-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:21.875022 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:21.875000    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/464b7491-c6a1-4927-95f2-f0bd4ff796ae-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:22.039164 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:22.039121    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:22.060734 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:23:22.060703    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5289e7a3_21ce_4848_beb1_ba5b0e212f81.slice/crio-6e6edd521c02a5b67ce8bac9014201e983f72d70404908c177831cd06d7bc609 WatchSource:0}: Error finding container 6e6edd521c02a5b67ce8bac9014201e983f72d70404908c177831cd06d7bc609: Status 404 returned error can't find the container with id 6e6edd521c02a5b67ce8bac9014201e983f72d70404908c177831cd06d7bc609
Sep 17 18:23:22.066060 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:22.066028    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"]
Sep 17 18:23:22.318094 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:22.318064    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"]
Sep 17 18:23:22.319744 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:23:22.319717    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464b7491_c6a1_4927_95f2_f0bd4ff796ae.slice/crio-9eea868e5cea3d3e258d2e4451168d953bd6a7c6a711c54d2156a1d3abce2f36 WatchSource:0}: Error finding container 9eea868e5cea3d3e258d2e4451168d953bd6a7c6a711c54d2156a1d3abce2f36: Status 404 returned error can't find the container with id 9eea868e5cea3d3e258d2e4451168d953bd6a7c6a711c54d2156a1d3abce2f36
Sep 17 18:23:22.334920 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:22.334892    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh" event={"ID":"464b7491-c6a1-4927-95f2-f0bd4ff796ae","Type":"ContainerStarted","Data":"9eea868e5cea3d3e258d2e4451168d953bd6a7c6a711c54d2156a1d3abce2f36"}
Sep 17 18:23:22.336991 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:22.336963    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh" event={"ID":"5289e7a3-21ce-4848-beb1-ba5b0e212f81","Type":"ContainerStarted","Data":"6f08acb9585bdfd23501f4003eb8f9ccd45ec343483c11d53d1b6cd4a59f9d14"}
Sep 17 18:23:22.337091 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:22.337018    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh" event={"ID":"5289e7a3-21ce-4848-beb1-ba5b0e212f81","Type":"ContainerStarted","Data":"6e6edd521c02a5b67ce8bac9014201e983f72d70404908c177831cd06d7bc609"}
Sep 17 18:23:23.344694 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:23.344648    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh" event={"ID":"464b7491-c6a1-4927-95f2-f0bd4ff796ae","Type":"ContainerStarted","Data":"bef5851cebf7b27019d8d21602c734695e25edf2a57f4dacf02593353124fd00"}
Sep 17 18:23:23.948654 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:23.948621    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:24.022128 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:24.022080    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:25.360442 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:25.360406    2574 generic.go:356] "Generic (PLEG): container finished" podID="464b7491-c6a1-4927-95f2-f0bd4ff796ae" containerID="bef5851cebf7b27019d8d21602c734695e25edf2a57f4dacf02593353124fd00" exitCode=0
Sep 17 18:23:25.360937 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:25.360477    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh" event={"ID":"464b7491-c6a1-4927-95f2-f0bd4ff796ae","Type":"ContainerDied","Data":"bef5851cebf7b27019d8d21602c734695e25edf2a57f4dacf02593353124fd00"}
Sep 17 18:23:25.362330 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:25.362300    2574 generic.go:356] "Generic (PLEG): container finished" podID="5289e7a3-21ce-4848-beb1-ba5b0e212f81" containerID="6f08acb9585bdfd23501f4003eb8f9ccd45ec343483c11d53d1b6cd4a59f9d14" exitCode=0
Sep 17 18:23:25.362477 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:25.362375    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh" event={"ID":"5289e7a3-21ce-4848-beb1-ba5b0e212f81","Type":"ContainerDied","Data":"6f08acb9585bdfd23501f4003eb8f9ccd45ec343483c11d53d1b6cd4a59f9d14"}
Sep 17 18:23:26.369815 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:26.369770    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh" event={"ID":"464b7491-c6a1-4927-95f2-f0bd4ff796ae","Type":"ContainerStarted","Data":"48029a0c2fc5abb9c020083be3d56f40f746bb8f9ad82e9a6c771c57818dc817"}
Sep 17 18:23:26.370237 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:26.370026    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:23:26.371989 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:26.371966    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh" event={"ID":"5289e7a3-21ce-4848-beb1-ba5b0e212f81","Type":"ContainerStarted","Data":"e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe"}
Sep 17 18:23:26.387191 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:26.387135    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh" podStartSLOduration=2.82889646 podStartE2EDuration="5.387119516s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.558223056s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:26.386389181 +0000 UTC m=+1576.687211499" watchObservedRunningTime="2026-09-17 18:23:26.387119516 +0000 UTC m=+1576.687941867"
Sep 17 18:23:26.406047 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:26.405983    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh" podStartSLOduration=2.831012502 podStartE2EDuration="5.405962466s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.574949964s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:26.4054297 +0000 UTC m=+1576.706252021" watchObservedRunningTime="2026-09-17 18:23:26.405962466 +0000 UTC m=+1576.706784791"
Sep 17 18:23:26.783057 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:26.783022    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:28.583608 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:28.583572    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:28.678052 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:28.678023    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:31.783652 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:31.783614    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:31.796677 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:31.796643    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:32.418880 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:32.418845    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:23:34.283933 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:34.283896    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:35.354917 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:35.354883    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:41.204006 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:41.203962    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:41.274045 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:41.274006    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:43.300826 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:43.300792    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:43.361476 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:43.361439    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:43.530232 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:43.530199    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:57.375113 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:57.375012    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:57.389072 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:23:57.389039    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:24:01.490783 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:01.490749    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh_464b7491-c6a1-4927-95f2-f0bd4ff796ae/main/0.log"
Sep 17 18:24:01.992328 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:01.992284    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:01.996010 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:01.995982    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"]
Sep 17 18:24:02.007185 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.007150    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"]
Sep 17 18:24:02.007463 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.007437    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh" podUID="464b7491-c6a1-4927-95f2-f0bd4ff796ae" containerName="main" containerID="cri-o://48029a0c2fc5abb9c020083be3d56f40f746bb8f9ad82e9a6c771c57818dc817" gracePeriod=30
Sep 17 18:24:02.008256 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:24:02.008031    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:24:02.008256 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:24:02.008256 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:24:02.008256 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe" cmd=["/bin/sleep","15"]
Sep 17 18:24:02.008256 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.008086    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh" message="" err=<
Sep 17 18:24:02.008256 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:24:02.008256 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:24:02.008256 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 18:24:02.008256 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:24:02.008104    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:24:02.008256 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:24:02.008256 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:24:02.008256 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh" podUID="5289e7a3-21ce-4848-beb1-ba5b0e212f81" containerName="main" containerID="cri-o://e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe"
Sep 17 18:24:02.008256 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.008145    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh" podUID="5289e7a3-21ce-4848-beb1-ba5b0e212f81" containerName="main" containerID="cri-o://e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe" gracePeriod=60
Sep 17 18:24:02.091231 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.091188    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:02.268756 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.268722    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:24:02.438139 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.438102    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-dshm\") pod \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") "
Sep 17 18:24:02.438356 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.438152    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-kserve-provision-location\") pod \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") "
Sep 17 18:24:02.438356 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.438177    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-model-cache\") pod \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") "
Sep 17 18:24:02.438356 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.438215    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tqzf\" (UniqueName: \"kubernetes.io/projected/5289e7a3-21ce-4848-beb1-ba5b0e212f81-kube-api-access-9tqzf\") pod \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") "
Sep 17 18:24:02.438356 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.438255    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-home\") pod \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") "
Sep 17 18:24:02.438356 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.438283    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-tmp-dir\") pod \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") "
Sep 17 18:24:02.438666 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.438531    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5289e7a3-21ce-4848-beb1-ba5b0e212f81-cabundle-cert\") pod \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") "
Sep 17 18:24:02.438666 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.438591    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5289e7a3-21ce-4848-beb1-ba5b0e212f81-tls-certs\") pod \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\" (UID: \"5289e7a3-21ce-4848-beb1-ba5b0e212f81\") "
Sep 17 18:24:02.438666 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.438608    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-home" (OuterVolumeSpecName: "home") pod "5289e7a3-21ce-4848-beb1-ba5b0e212f81" (UID: "5289e7a3-21ce-4848-beb1-ba5b0e212f81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:02.438666 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.438626    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-model-cache" (OuterVolumeSpecName: "model-cache") pod "5289e7a3-21ce-4848-beb1-ba5b0e212f81" (UID: "5289e7a3-21ce-4848-beb1-ba5b0e212f81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:02.439025 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.438837    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5289e7a3-21ce-4848-beb1-ba5b0e212f81" (UID: "5289e7a3-21ce-4848-beb1-ba5b0e212f81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:02.439025 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.438941    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.439025 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.438959    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.439276 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.439021    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5289e7a3-21ce-4848-beb1-ba5b0e212f81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5289e7a3-21ce-4848-beb1-ba5b0e212f81" (UID: "5289e7a3-21ce-4848-beb1-ba5b0e212f81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:02.439276 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.439235    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.441162 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.441119    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-dshm" (OuterVolumeSpecName: "dshm") pod "5289e7a3-21ce-4848-beb1-ba5b0e212f81" (UID: "5289e7a3-21ce-4848-beb1-ba5b0e212f81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:02.441162 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.441156    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5289e7a3-21ce-4848-beb1-ba5b0e212f81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5289e7a3-21ce-4848-beb1-ba5b0e212f81" (UID: "5289e7a3-21ce-4848-beb1-ba5b0e212f81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:02.441345 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.441321    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5289e7a3-21ce-4848-beb1-ba5b0e212f81-kube-api-access-9tqzf" (OuterVolumeSpecName: "kube-api-access-9tqzf") pod "5289e7a3-21ce-4848-beb1-ba5b0e212f81" (UID: "5289e7a3-21ce-4848-beb1-ba5b0e212f81"). InnerVolumeSpecName "kube-api-access-9tqzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:02.462177 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.462136    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5289e7a3-21ce-4848-beb1-ba5b0e212f81" (UID: "5289e7a3-21ce-4848-beb1-ba5b0e212f81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:02.540421 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.540383    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5289e7a3-21ce-4848-beb1-ba5b0e212f81-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.540421 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.540417    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5289e7a3-21ce-4848-beb1-ba5b0e212f81-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.540421 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.540426    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.541139 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.540435    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5289e7a3-21ce-4848-beb1-ba5b0e212f81-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.541139 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.540448    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tqzf\" (UniqueName: \"kubernetes.io/projected/5289e7a3-21ce-4848-beb1-ba5b0e212f81-kube-api-access-9tqzf\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.584755 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.584714    2574 generic.go:356] "Generic (PLEG): container finished" podID="5289e7a3-21ce-4848-beb1-ba5b0e212f81" containerID="e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe" exitCode=0
Sep 17 18:24:02.584881 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.584791    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"
Sep 17 18:24:02.584881 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.584790    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh" event={"ID":"5289e7a3-21ce-4848-beb1-ba5b0e212f81","Type":"ContainerDied","Data":"e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe"}
Sep 17 18:24:02.585008 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.584908    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh" event={"ID":"5289e7a3-21ce-4848-beb1-ba5b0e212f81","Type":"ContainerDied","Data":"6e6edd521c02a5b67ce8bac9014201e983f72d70404908c177831cd06d7bc609"}
Sep 17 18:24:02.585008 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.584941    2574 scope.go:117] "RemoveContainer" containerID="e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe"
Sep 17 18:24:02.587352 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.587320    2574 generic.go:356] "Generic (PLEG): container finished" podID="464b7491-c6a1-4927-95f2-f0bd4ff796ae" containerID="48029a0c2fc5abb9c020083be3d56f40f746bb8f9ad82e9a6c771c57818dc817" exitCode=0
Sep 17 18:24:02.587504 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.587392    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh" event={"ID":"464b7491-c6a1-4927-95f2-f0bd4ff796ae","Type":"ContainerDied","Data":"48029a0c2fc5abb9c020083be3d56f40f746bb8f9ad82e9a6c771c57818dc817"}
Sep 17 18:24:02.596867 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.596843    2574 scope.go:117] "RemoveContainer" containerID="6f08acb9585bdfd23501f4003eb8f9ccd45ec343483c11d53d1b6cd4a59f9d14"
Sep 17 18:24:02.610205 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.610151    2574 scope.go:117] "RemoveContainer" containerID="e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe"
Sep 17 18:24:02.611042 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:24:02.611014    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe\": container with ID starting with e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe not found: ID does not exist" containerID="e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe"
Sep 17 18:24:02.611159 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.611054    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe"} err="failed to get container status \"e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe\": rpc error: code = NotFound desc = could not find container \"e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe\": container with ID starting with e5414fff4fff3772ed435293259fca044eedd599f0d63d8dd6d45e41c298defe not found: ID does not exist"
Sep 17 18:24:02.611159 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.611080    2574 scope.go:117] "RemoveContainer" containerID="6f08acb9585bdfd23501f4003eb8f9ccd45ec343483c11d53d1b6cd4a59f9d14"
Sep 17 18:24:02.611436 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.611416    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"]
Sep 17 18:24:02.611580 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:24:02.611552    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f08acb9585bdfd23501f4003eb8f9ccd45ec343483c11d53d1b6cd4a59f9d14\": container with ID starting with 6f08acb9585bdfd23501f4003eb8f9ccd45ec343483c11d53d1b6cd4a59f9d14 not found: ID does not exist" containerID="6f08acb9585bdfd23501f4003eb8f9ccd45ec343483c11d53d1b6cd4a59f9d14"
Sep 17 18:24:02.611650 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.611591    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f08acb9585bdfd23501f4003eb8f9ccd45ec343483c11d53d1b6cd4a59f9d14"} err="failed to get container status \"6f08acb9585bdfd23501f4003eb8f9ccd45ec343483c11d53d1b6cd4a59f9d14\": rpc error: code = NotFound desc = could not find container \"6f08acb9585bdfd23501f4003eb8f9ccd45ec343483c11d53d1b6cd4a59f9d14\": container with ID starting with 6f08acb9585bdfd23501f4003eb8f9ccd45ec343483c11d53d1b6cd4a59f9d14 not found: ID does not exist"
Sep 17 18:24:02.613873 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.613850    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-4s5gh"]
Sep 17 18:24:02.661198 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.661165    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:24:02.843226 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.843128    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/464b7491-c6a1-4927-95f2-f0bd4ff796ae-kserve-provision-location\") pod \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") "
Sep 17 18:24:02.843226 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.843208    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/464b7491-c6a1-4927-95f2-f0bd4ff796ae-tls-certs\") pod \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") "
Sep 17 18:24:02.843476 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.843232    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/464b7491-c6a1-4927-95f2-f0bd4ff796ae-cabundle-cert\") pod \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") "
Sep 17 18:24:02.843476 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.843246    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464b7491-c6a1-4927-95f2-f0bd4ff796ae-tmp-dir\") pod \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\" (UID: \"464b7491-c6a1-4927-95f2-f0bd4ff796ae\") "
Sep 17 18:24:02.843676 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.843649    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464b7491-c6a1-4927-95f2-f0bd4ff796ae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "464b7491-c6a1-4927-95f2-f0bd4ff796ae" (UID: "464b7491-c6a1-4927-95f2-f0bd4ff796ae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:02.843811 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.843788    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464b7491-c6a1-4927-95f2-f0bd4ff796ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "464b7491-c6a1-4927-95f2-f0bd4ff796ae" (UID: "464b7491-c6a1-4927-95f2-f0bd4ff796ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:02.845751 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.845731    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464b7491-c6a1-4927-95f2-f0bd4ff796ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "464b7491-c6a1-4927-95f2-f0bd4ff796ae" (UID: "464b7491-c6a1-4927-95f2-f0bd4ff796ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:02.867501 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.867461    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464b7491-c6a1-4927-95f2-f0bd4ff796ae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "464b7491-c6a1-4927-95f2-f0bd4ff796ae" (UID: "464b7491-c6a1-4927-95f2-f0bd4ff796ae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:02.943995 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.943955    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/464b7491-c6a1-4927-95f2-f0bd4ff796ae-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.943995 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.943986    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/464b7491-c6a1-4927-95f2-f0bd4ff796ae-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.943995 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.943998    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464b7491-c6a1-4927-95f2-f0bd4ff796ae-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.943995 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:02.944006    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/464b7491-c6a1-4927-95f2-f0bd4ff796ae-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:24:03.594919 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:03.594880    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"
Sep 17 18:24:03.594919 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:03.594903    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh" event={"ID":"464b7491-c6a1-4927-95f2-f0bd4ff796ae","Type":"ContainerDied","Data":"9eea868e5cea3d3e258d2e4451168d953bd6a7c6a711c54d2156a1d3abce2f36"}
Sep 17 18:24:03.595485 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:03.594976    2574 scope.go:117] "RemoveContainer" containerID="48029a0c2fc5abb9c020083be3d56f40f746bb8f9ad82e9a6c771c57818dc817"
Sep 17 18:24:03.605296 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:03.605268    2574 scope.go:117] "RemoveContainer" containerID="bef5851cebf7b27019d8d21602c734695e25edf2a57f4dacf02593353124fd00"
Sep 17 18:24:03.617430 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:03.617402    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"]
Sep 17 18:24:03.622334 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:03.622290    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6b5cfb859b-pcxjh"]
Sep 17 18:24:03.939225 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:03.939142    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="464b7491-c6a1-4927-95f2-f0bd4ff796ae" path="/var/lib/kubelet/pods/464b7491-c6a1-4927-95f2-f0bd4ff796ae/volumes"
Sep 17 18:24:03.939631 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:03.939617    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5289e7a3-21ce-4848-beb1-ba5b0e212f81" path="/var/lib/kubelet/pods/5289e7a3-21ce-4848-beb1-ba5b0e212f81/volumes"
Sep 17 18:24:13.979422 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:13.979370    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:14.364097 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:14.364060    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:25.359140 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:25.359095    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:51.819692 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:51.819659    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:51.918819 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:51.918780    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:58.414979 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:24:58.414937    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:00.016713 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.016677    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:00.667974 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.667934    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"]
Sep 17 18:25:00.668354 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.668331    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464b7491-c6a1-4927-95f2-f0bd4ff796ae" containerName="storage-initializer"
Sep 17 18:25:00.668354 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.668359    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="464b7491-c6a1-4927-95f2-f0bd4ff796ae" containerName="storage-initializer"
Sep 17 18:25:00.668620 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.668378    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5289e7a3-21ce-4848-beb1-ba5b0e212f81" containerName="main"
Sep 17 18:25:00.668620 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.668386    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5289e7a3-21ce-4848-beb1-ba5b0e212f81" containerName="main"
Sep 17 18:25:00.668620 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.668406    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5289e7a3-21ce-4848-beb1-ba5b0e212f81" containerName="storage-initializer"
Sep 17 18:25:00.668620 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.668415    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5289e7a3-21ce-4848-beb1-ba5b0e212f81" containerName="storage-initializer"
Sep 17 18:25:00.668620 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.668432    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464b7491-c6a1-4927-95f2-f0bd4ff796ae" containerName="main"
Sep 17 18:25:00.668620 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.668446    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="464b7491-c6a1-4927-95f2-f0bd4ff796ae" containerName="main"
Sep 17 18:25:00.668620 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.668548    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5289e7a3-21ce-4848-beb1-ba5b0e212f81" containerName="main"
Sep 17 18:25:00.668620 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.668562    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="464b7491-c6a1-4927-95f2-f0bd4ff796ae" containerName="main"
Sep 17 18:25:00.671725 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.671702    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.674985 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.674959    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 18:25:00.675113 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.675007    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 18:25:00.675113 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.674961    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 18:25:00.675113 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.674961    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-8kqck\""
Sep 17 18:25:00.682080 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.682054    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"]
Sep 17 18:25:00.757088 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.757048    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:00.827462 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.827422    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.827674 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.827485    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc70453-0526-4716-8ec0-ce20abeb36fb-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.827674 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.827580    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.827674 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.827603    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g7vk\" (UniqueName: \"kubernetes.io/projected/1cc70453-0526-4716-8ec0-ce20abeb36fb-kube-api-access-5g7vk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.827674 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.827621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.827674 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.827640    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.928794 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.928705    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.928794 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.928755    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.929024 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.928797    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc70453-0526-4716-8ec0-ce20abeb36fb-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.929024 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.928827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.929024 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.928845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g7vk\" (UniqueName: \"kubernetes.io/projected/1cc70453-0526-4716-8ec0-ce20abeb36fb-kube-api-access-5g7vk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.929024 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.928870    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.929246 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.929176    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.929297 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.929239    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.929297 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.929273    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.931298 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.931279    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.931585 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.931570    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc70453-0526-4716-8ec0-ce20abeb36fb-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.936729 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.936706    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g7vk\" (UniqueName: \"kubernetes.io/projected/1cc70453-0526-4716-8ec0-ce20abeb36fb-kube-api-access-5g7vk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.984899 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.984862    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:00.988740 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.988716    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"]
Sep 17 18:25:00.992213 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.992179    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"
Sep 17 18:25:00.994489 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:00.994464    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-dlfkl\""
Sep 17 18:25:01.002701 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.002676    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"]
Sep 17 18:25:01.030226 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.030179    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:01.130134 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.130093    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tmxm\" (UniqueName: \"kubernetes.io/projected/19c6f974-a007-4feb-bd45-b2fe978894ed-kube-api-access-9tmxm\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767\" (UID: \"19c6f974-a007-4feb-bd45-b2fe978894ed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"
Sep 17 18:25:01.130334 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.130149    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c6f974-a007-4feb-bd45-b2fe978894ed-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767\" (UID: \"19c6f974-a007-4feb-bd45-b2fe978894ed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"
Sep 17 18:25:01.230834 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.230739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tmxm\" (UniqueName: \"kubernetes.io/projected/19c6f974-a007-4feb-bd45-b2fe978894ed-kube-api-access-9tmxm\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767\" (UID: \"19c6f974-a007-4feb-bd45-b2fe978894ed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"
Sep 17 18:25:01.230834 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.230784    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c6f974-a007-4feb-bd45-b2fe978894ed-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767\" (UID: \"19c6f974-a007-4feb-bd45-b2fe978894ed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"
Sep 17 18:25:01.233761 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.233725    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c6f974-a007-4feb-bd45-b2fe978894ed-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767\" (UID: \"19c6f974-a007-4feb-bd45-b2fe978894ed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"
Sep 17 18:25:01.239178 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.239149    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tmxm\" (UniqueName: \"kubernetes.io/projected/19c6f974-a007-4feb-bd45-b2fe978894ed-kube-api-access-9tmxm\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767\" (UID: \"19c6f974-a007-4feb-bd45-b2fe978894ed\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"
Sep 17 18:25:01.261228 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.261202    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"]
Sep 17 18:25:01.263213 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:25:01.263175    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc70453_0526_4716_8ec0_ce20abeb36fb.slice/crio-23055f7f74874ef6a4a79ec635e84f3f302293c9586c9663e9773604c1920ea5 WatchSource:0}: Error finding container 23055f7f74874ef6a4a79ec635e84f3f302293c9586c9663e9773604c1920ea5: Status 404 returned error can't find the container with id 23055f7f74874ef6a4a79ec635e84f3f302293c9586c9663e9773604c1920ea5
Sep 17 18:25:01.327675 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.327649    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"
Sep 17 18:25:01.603203 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.603167    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"]
Sep 17 18:25:01.604075 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:25:01.604039    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c6f974_a007_4feb_bd45_b2fe978894ed.slice/crio-cf6395af8e6ea507e533baecb4168d5c0bf733dffaf3eae686124ba022525a06 WatchSource:0}: Error finding container cf6395af8e6ea507e533baecb4168d5c0bf733dffaf3eae686124ba022525a06: Status 404 returned error can't find the container with id cf6395af8e6ea507e533baecb4168d5c0bf733dffaf3eae686124ba022525a06
Sep 17 18:25:01.922647 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.922557    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf" event={"ID":"1cc70453-0526-4716-8ec0-ce20abeb36fb","Type":"ContainerStarted","Data":"c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124"}
Sep 17 18:25:01.922647 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.922620    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf" event={"ID":"1cc70453-0526-4716-8ec0-ce20abeb36fb","Type":"ContainerStarted","Data":"23055f7f74874ef6a4a79ec635e84f3f302293c9586c9663e9773604c1920ea5"}
Sep 17 18:25:01.924160 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.924135    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767" event={"ID":"19c6f974-a007-4feb-bd45-b2fe978894ed","Type":"ContainerStarted","Data":"26c71e2e50b24e9c1232c3edf1b744c0e3daa89e4c95af3ee00410696c1c800c"}
Sep 17 18:25:01.924276 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.924177    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767" event={"ID":"19c6f974-a007-4feb-bd45-b2fe978894ed","Type":"ContainerStarted","Data":"cf6395af8e6ea507e533baecb4168d5c0bf733dffaf3eae686124ba022525a06"}
Sep 17 18:25:01.924276 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.924198    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"
Sep 17 18:25:01.945024 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.944972    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf" podStartSLOduration=1.9449552319999999 podStartE2EDuration="1.944955232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:25:01.942889606 +0000 UTC m=+1672.243711930" watchObservedRunningTime="2026-09-17 18:25:01.944955232 +0000 UTC m=+1672.245777556"
Sep 17 18:25:01.960140 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.960077    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767" podStartSLOduration=1.960056712 podStartE2EDuration="1.960056712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:25:01.959113365 +0000 UTC m=+1672.259935689" watchObservedRunningTime="2026-09-17 18:25:01.960056712 +0000 UTC m=+1672.260879104"
Sep 17 18:25:01.985114 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:01.985080    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:04.450995 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:04.450957    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:04.525449 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:04.525414    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:10.985590 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:10.985543    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:10.998877 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:10.998845    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:11.407126 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:11.407096    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:11.992799 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:11.992766    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:32.932620 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:32.932500    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"
Sep 17 18:25:34.464390 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.464343    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"]
Sep 17 18:25:34.467303 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.467273    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:34.473695 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.473660    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"]
Sep 17 18:25:34.475277 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:25:34.475240    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:25:34.475277 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:25:34.475277 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:25:34.475277 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124" cmd=["/bin/sleep","15"]
Sep 17 18:25:34.475678 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.475299    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf" message="" err=<
Sep 17 18:25:34.475678 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:25:34.475678 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:25:34.475678 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 18:25:34.475678 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:25:34.475316    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:25:34.475678 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:25:34.475678 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:25:34.475678 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf" podUID="1cc70453-0526-4716-8ec0-ce20abeb36fb" containerName="main" containerID="cri-o://c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124"
Sep 17 18:25:34.475678 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.475358    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf" podUID="1cc70453-0526-4716-8ec0-ce20abeb36fb" containerName="main" containerID="cri-o://c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124" gracePeriod=60
Sep 17 18:25:34.523710 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.523677    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:34.832715 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.832690    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:34.931166 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.931062    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-dshm\") pod \"1cc70453-0526-4716-8ec0-ce20abeb36fb\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") "
Sep 17 18:25:34.931365 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.931220    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-home\") pod \"1cc70453-0526-4716-8ec0-ce20abeb36fb\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") "
Sep 17 18:25:34.931365 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.931257    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-tmp-dir\") pod \"1cc70453-0526-4716-8ec0-ce20abeb36fb\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") "
Sep 17 18:25:34.931365 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.931284    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g7vk\" (UniqueName: \"kubernetes.io/projected/1cc70453-0526-4716-8ec0-ce20abeb36fb-kube-api-access-5g7vk\") pod \"1cc70453-0526-4716-8ec0-ce20abeb36fb\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") "
Sep 17 18:25:34.931365 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.931317    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-model-cache\") pod \"1cc70453-0526-4716-8ec0-ce20abeb36fb\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") "
Sep 17 18:25:34.931365 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.931360    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc70453-0526-4716-8ec0-ce20abeb36fb-tls-certs\") pod \"1cc70453-0526-4716-8ec0-ce20abeb36fb\" (UID: \"1cc70453-0526-4716-8ec0-ce20abeb36fb\") "
Sep 17 18:25:34.931673 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.931589    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-home" (OuterVolumeSpecName: "home") pod "1cc70453-0526-4716-8ec0-ce20abeb36fb" (UID: "1cc70453-0526-4716-8ec0-ce20abeb36fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:34.931673 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.931600    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1cc70453-0526-4716-8ec0-ce20abeb36fb" (UID: "1cc70453-0526-4716-8ec0-ce20abeb36fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:34.931673 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.931610    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "1cc70453-0526-4716-8ec0-ce20abeb36fb" (UID: "1cc70453-0526-4716-8ec0-ce20abeb36fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:34.931817 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.931682    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:25:34.933543 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.933498    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-dshm" (OuterVolumeSpecName: "dshm") pod "1cc70453-0526-4716-8ec0-ce20abeb36fb" (UID: "1cc70453-0526-4716-8ec0-ce20abeb36fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:34.933655 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.933584    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc70453-0526-4716-8ec0-ce20abeb36fb-kube-api-access-5g7vk" (OuterVolumeSpecName: "kube-api-access-5g7vk") pod "1cc70453-0526-4716-8ec0-ce20abeb36fb" (UID: "1cc70453-0526-4716-8ec0-ce20abeb36fb"). InnerVolumeSpecName "kube-api-access-5g7vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:34.933655 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:34.933637    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc70453-0526-4716-8ec0-ce20abeb36fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cc70453-0526-4716-8ec0-ce20abeb36fb" (UID: "1cc70453-0526-4716-8ec0-ce20abeb36fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:35.032310 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.032275    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:25:35.032310 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.032305    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g7vk\" (UniqueName: \"kubernetes.io/projected/1cc70453-0526-4716-8ec0-ce20abeb36fb-kube-api-access-5g7vk\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:25:35.032310 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.032316    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:25:35.032580 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.032324    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc70453-0526-4716-8ec0-ce20abeb36fb-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:25:35.032580 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.032333    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc70453-0526-4716-8ec0-ce20abeb36fb-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:25:35.105743 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.105703    2574 generic.go:356] "Generic (PLEG): container finished" podID="1cc70453-0526-4716-8ec0-ce20abeb36fb" containerID="c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124" exitCode=0
Sep 17 18:25:35.105919 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.105757    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"
Sep 17 18:25:35.105919 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.105764    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf" event={"ID":"1cc70453-0526-4716-8ec0-ce20abeb36fb","Type":"ContainerDied","Data":"c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124"}
Sep 17 18:25:35.105919 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.105810    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf" event={"ID":"1cc70453-0526-4716-8ec0-ce20abeb36fb","Type":"ContainerDied","Data":"23055f7f74874ef6a4a79ec635e84f3f302293c9586c9663e9773604c1920ea5"}
Sep 17 18:25:35.105919 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.105830    2574 scope.go:117] "RemoveContainer" containerID="c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124"
Sep 17 18:25:35.115616 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.115593    2574 scope.go:117] "RemoveContainer" containerID="c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124"
Sep 17 18:25:35.115905 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:25:35.115885    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124\": container with ID starting with c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124 not found: ID does not exist" containerID="c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124"
Sep 17 18:25:35.115981 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.115911    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124"} err="failed to get container status \"c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124\": rpc error: code = NotFound desc = could not find container \"c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124\": container with ID starting with c705d19a8ea4f14bed5e1d6894de15e8c829700555945702f8945255dcb61124 not found: ID does not exist"
Sep 17 18:25:35.127370 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.127344    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"]
Sep 17 18:25:35.131024 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.131002    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ptcqf"]
Sep 17 18:25:35.938841 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:35.938804    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc70453-0526-4716-8ec0-ce20abeb36fb" path="/var/lib/kubelet/pods/1cc70453-0526-4716-8ec0-ce20abeb36fb/volumes"
Sep 17 18:25:42.932194 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:42.932144    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767" podUID="19c6f974-a007-4feb-bd45-b2fe978894ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:25:48.291976 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:48.291928    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:48.351403 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:48.351370    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:49.494622 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:49.494554    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767" podUID="19c6f974-a007-4feb-bd45-b2fe978894ed" containerName="main" containerID="cri-o://26c71e2e50b24e9c1232c3edf1b744c0e3daa89e4c95af3ee00410696c1c800c" gracePeriod=45
Sep 17 18:25:52.932122 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:25:52.932074    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767" podUID="19c6f974-a007-4feb-bd45-b2fe978894ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:26:02.932747 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:02.932700    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767" podUID="19c6f974-a007-4feb-bd45-b2fe978894ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:26:02.933178 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:02.932848    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"
Sep 17 18:26:12.932422 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:12.932379    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767" podUID="19c6f974-a007-4feb-bd45-b2fe978894ed" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:26:19.756743 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:19.756674    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"
Sep 17 18:26:19.827215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:19.827178    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tmxm\" (UniqueName: \"kubernetes.io/projected/19c6f974-a007-4feb-bd45-b2fe978894ed-kube-api-access-9tmxm\") pod \"19c6f974-a007-4feb-bd45-b2fe978894ed\" (UID: \"19c6f974-a007-4feb-bd45-b2fe978894ed\") "
Sep 17 18:26:19.827421 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:19.827248    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c6f974-a007-4feb-bd45-b2fe978894ed-tls-certs\") pod \"19c6f974-a007-4feb-bd45-b2fe978894ed\" (UID: \"19c6f974-a007-4feb-bd45-b2fe978894ed\") "
Sep 17 18:26:19.829705 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:19.829668    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c6f974-a007-4feb-bd45-b2fe978894ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19c6f974-a007-4feb-bd45-b2fe978894ed" (UID: "19c6f974-a007-4feb-bd45-b2fe978894ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:19.829822 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:19.829702    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c6f974-a007-4feb-bd45-b2fe978894ed-kube-api-access-9tmxm" (OuterVolumeSpecName: "kube-api-access-9tmxm") pod "19c6f974-a007-4feb-bd45-b2fe978894ed" (UID: "19c6f974-a007-4feb-bd45-b2fe978894ed"). InnerVolumeSpecName "kube-api-access-9tmxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:19.928835 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:19.928795    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tmxm\" (UniqueName: \"kubernetes.io/projected/19c6f974-a007-4feb-bd45-b2fe978894ed-kube-api-access-9tmxm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:26:19.928835 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:19.928827    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19c6f974-a007-4feb-bd45-b2fe978894ed-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:26:20.353232 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:20.353196    2574 generic.go:356] "Generic (PLEG): container finished" podID="19c6f974-a007-4feb-bd45-b2fe978894ed" containerID="26c71e2e50b24e9c1232c3edf1b744c0e3daa89e4c95af3ee00410696c1c800c" exitCode=0
Sep 17 18:26:20.353471 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:20.353241    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"
Sep 17 18:26:20.353471 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:20.353286    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767" event={"ID":"19c6f974-a007-4feb-bd45-b2fe978894ed","Type":"ContainerDied","Data":"26c71e2e50b24e9c1232c3edf1b744c0e3daa89e4c95af3ee00410696c1c800c"}
Sep 17 18:26:20.353471 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:20.353343    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767" event={"ID":"19c6f974-a007-4feb-bd45-b2fe978894ed","Type":"ContainerDied","Data":"cf6395af8e6ea507e533baecb4168d5c0bf733dffaf3eae686124ba022525a06"}
Sep 17 18:26:20.353471 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:20.353374    2574 scope.go:117] "RemoveContainer" containerID="26c71e2e50b24e9c1232c3edf1b744c0e3daa89e4c95af3ee00410696c1c800c"
Sep 17 18:26:20.362769 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:20.362752    2574 scope.go:117] "RemoveContainer" containerID="26c71e2e50b24e9c1232c3edf1b744c0e3daa89e4c95af3ee00410696c1c800c"
Sep 17 18:26:20.363054 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:26:20.363032    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c71e2e50b24e9c1232c3edf1b744c0e3daa89e4c95af3ee00410696c1c800c\": container with ID starting with 26c71e2e50b24e9c1232c3edf1b744c0e3daa89e4c95af3ee00410696c1c800c not found: ID does not exist" containerID="26c71e2e50b24e9c1232c3edf1b744c0e3daa89e4c95af3ee00410696c1c800c"
Sep 17 18:26:20.363101 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:20.363063    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c71e2e50b24e9c1232c3edf1b744c0e3daa89e4c95af3ee00410696c1c800c"} err="failed to get container status \"26c71e2e50b24e9c1232c3edf1b744c0e3daa89e4c95af3ee00410696c1c800c\": rpc error: code = NotFound desc = could not find container \"26c71e2e50b24e9c1232c3edf1b744c0e3daa89e4c95af3ee00410696c1c800c\": container with ID starting with 26c71e2e50b24e9c1232c3edf1b744c0e3daa89e4c95af3ee00410696c1c800c not found: ID does not exist"
Sep 17 18:26:20.370827 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:20.370802    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"]
Sep 17 18:26:20.375916 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:20.375894    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sched6767"]
Sep 17 18:26:21.938842 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:21.938809    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19c6f974-a007-4feb-bd45-b2fe978894ed" path="/var/lib/kubelet/pods/19c6f974-a007-4feb-bd45-b2fe978894ed/volumes"
Sep 17 18:26:24.561776 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:24.561729    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:24.656851 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:24.656812    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:30.830033 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:30.829997    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:36.600201 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:36.600159    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:36.681271 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:36.681228    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:38.422456 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:38.422393    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:38.520828 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:38.520789    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:38.649095 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:38.649062    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:38.764844 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:38.764815    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:42.292921 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:42.292889    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:45.101391 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:45.101350    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:51.117995 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.117913    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:51.219190 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.219153    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:51.565979 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.565938    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"]
Sep 17 18:26:51.566353 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.566337    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c6f974-a007-4feb-bd45-b2fe978894ed" containerName="main"
Sep 17 18:26:51.566353 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.566354    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c6f974-a007-4feb-bd45-b2fe978894ed" containerName="main"
Sep 17 18:26:51.566497 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.566378    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc70453-0526-4716-8ec0-ce20abeb36fb" containerName="main"
Sep 17 18:26:51.566497 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.566385    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc70453-0526-4716-8ec0-ce20abeb36fb" containerName="main"
Sep 17 18:26:51.566497 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.566452    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c6f974-a007-4feb-bd45-b2fe978894ed" containerName="main"
Sep 17 18:26:51.566497 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.566463    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc70453-0526-4716-8ec0-ce20abeb36fb" containerName="main"
Sep 17 18:26:51.569568 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.569543    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.572234 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.572208    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 18:26:51.573279 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.573255    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-b9n8s\""
Sep 17 18:26:51.573279 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.573270    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 18:26:51.573421 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.573256    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 18:26:51.580586 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.580564    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"]
Sep 17 18:26:51.597137 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.597074    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:51.710702 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.710664    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwscd\" (UniqueName: \"kubernetes.io/projected/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-kube-api-access-xwscd\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.710877 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.710709    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.710877 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.710758    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.710877 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.710814    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.710877 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.710865    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.711037 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.710925    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.807046 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.807004    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:51.811722 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.811696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.811877 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.811763    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.811877 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.811799    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwscd\" (UniqueName: \"kubernetes.io/projected/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-kube-api-access-xwscd\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.811877 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.811833    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.811877 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.811857    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.812099 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.811903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.812161 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.812133    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.812247 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.812219    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.812359 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.812344    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.814277 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.814253    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.814685 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.814667    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.819587 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.819539    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwscd\" (UniqueName: \"kubernetes.io/projected/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-kube-api-access-xwscd\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:51.881479 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:51.881440    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:26:52.160012 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:52.159981    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"]
Sep 17 18:26:52.161705 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:26:52.161677    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38cfe3a3_4e9a_4946_a15a_fd6cdca90e6a.slice/crio-04926c2dbd2df61c5d323d55e3cbf5a687caf0edd8bc87d652e5d3b163fafb65 WatchSource:0}: Error finding container 04926c2dbd2df61c5d323d55e3cbf5a687caf0edd8bc87d652e5d3b163fafb65: Status 404 returned error can't find the container with id 04926c2dbd2df61c5d323d55e3cbf5a687caf0edd8bc87d652e5d3b163fafb65
Sep 17 18:26:52.523491 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:52.523453    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn" event={"ID":"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a","Type":"ContainerStarted","Data":"2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2"}
Sep 17 18:26:52.523684 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:52.523526    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn" event={"ID":"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a","Type":"ContainerStarted","Data":"04926c2dbd2df61c5d323d55e3cbf5a687caf0edd8bc87d652e5d3b163fafb65"}
Sep 17 18:26:52.540346 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:52.540291    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn" podStartSLOduration=1.540275649 podStartE2EDuration="1.540275649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:52.538435632 +0000 UTC m=+1782.839257971" watchObservedRunningTime="2026-09-17 18:26:52.540275649 +0000 UTC m=+1782.841097973"
Sep 17 18:26:52.881715 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:26:52.881618    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:27:01.881865 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:01.881824    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:27:01.894661 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:01.894632    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:27:02.584336 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:02.584306    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:27:03.269313 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:03.269273    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:11.931141 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:11.931105    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:27:11.940831 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:11.940807    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:27:15.779720 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:15.779682    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:48.462357 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.462313    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"]
Sep 17 18:27:48.465925 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.465898    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.468271 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.468241    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 18:27:48.474662 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.474638    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"]
Sep 17 18:27:48.553995 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.553958    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:48.581828 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.581791    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.582018 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.581842    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.582018 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.581872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkwxk\" (UniqueName: \"kubernetes.io/projected/64ef4b97-a782-41b6-82d5-511179198afd-kube-api-access-mkwxk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.582018 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.581914    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ef4b97-a782-41b6-82d5-511179198afd-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.582018 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.581978    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.582018 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.582002    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.682986 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.682945    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.683198 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.683000    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.683198 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.683047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.683198 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.683086    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.683198 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.683114    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwxk\" (UniqueName: \"kubernetes.io/projected/64ef4b97-a782-41b6-82d5-511179198afd-kube-api-access-mkwxk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.683198 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.683153    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ef4b97-a782-41b6-82d5-511179198afd-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.683470 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.683415    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.683470 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.683425    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.683610 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.683542    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.685561 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.685539    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.685907 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.685887    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ef4b97-a782-41b6-82d5-511179198afd-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.691730 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.691709    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkwxk\" (UniqueName: \"kubernetes.io/projected/64ef4b97-a782-41b6-82d5-511179198afd-kube-api-access-mkwxk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.777313 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.777275    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:48.820300 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:48.820269    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:49.061562 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:49.061533    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"]
Sep 17 18:27:49.063919 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:27:49.063888    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ef4b97_a782_41b6_82d5_511179198afd.slice/crio-4f60a6eebc7ea8b86ff96b394c3a600d29845675072704ba663c9d6890f19064 WatchSource:0}: Error finding container 4f60a6eebc7ea8b86ff96b394c3a600d29845675072704ba663c9d6890f19064: Status 404 returned error can't find the container with id 4f60a6eebc7ea8b86ff96b394c3a600d29845675072704ba663c9d6890f19064
Sep 17 18:27:49.837389 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:49.837342    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9" event={"ID":"64ef4b97-a782-41b6-82d5-511179198afd","Type":"ContainerStarted","Data":"71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3"}
Sep 17 18:27:49.837914 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:49.837407    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9" event={"ID":"64ef4b97-a782-41b6-82d5-511179198afd","Type":"ContainerStarted","Data":"4f60a6eebc7ea8b86ff96b394c3a600d29845675072704ba663c9d6890f19064"}
Sep 17 18:27:49.855247 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:49.855189    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9" podStartSLOduration=1.855170619 podStartE2EDuration="1.855170619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:49.854346658 +0000 UTC m=+1840.155168983" watchObservedRunningTime="2026-09-17 18:27:49.855170619 +0000 UTC m=+1840.155992943"
Sep 17 18:27:50.778306 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:50.778257    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:58.778147 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:58.778096    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:58.791067 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:58.791034    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:27:58.909523 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:27:58.909485    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:28:10.886920 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:10.886877    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:13.835461 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:13.835429    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:16.099595 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:16.099545    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:16.197368 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:16.197318    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:21.135322 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.135229    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:21.309568 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.309536    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:21.571906 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.571865    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"]
Sep 17 18:28:21.576065 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.576037    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"
Sep 17 18:28:21.580237 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.580210    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 18:28:21.580389 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.580270    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-srtpr\""
Sep 17 18:28:21.580389 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.580210    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-4gr2d\""
Sep 17 18:28:21.580546 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.580210    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 18:28:21.580546 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.580210    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 18:28:21.584188 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.584162    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"]
Sep 17 18:28:21.599992 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.599962    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:21.679218 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.679183    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826e2da1-89ab-4043-b141-3a2c9798379e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8\" (UID: \"826e2da1-89ab-4043-b141-3a2c9798379e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"
Sep 17 18:28:21.679403 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.679277    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlh8c\" (UniqueName: \"kubernetes.io/projected/826e2da1-89ab-4043-b141-3a2c9798379e-kube-api-access-mlh8c\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8\" (UID: \"826e2da1-89ab-4043-b141-3a2c9798379e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"
Sep 17 18:28:21.780286 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.780236    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826e2da1-89ab-4043-b141-3a2c9798379e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8\" (UID: \"826e2da1-89ab-4043-b141-3a2c9798379e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"
Sep 17 18:28:21.780494 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.780359    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlh8c\" (UniqueName: \"kubernetes.io/projected/826e2da1-89ab-4043-b141-3a2c9798379e-kube-api-access-mlh8c\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8\" (UID: \"826e2da1-89ab-4043-b141-3a2c9798379e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"
Sep 17 18:28:21.783202 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.783174    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826e2da1-89ab-4043-b141-3a2c9798379e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8\" (UID: \"826e2da1-89ab-4043-b141-3a2c9798379e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"
Sep 17 18:28:21.787958 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.787926    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlh8c\" (UniqueName: \"kubernetes.io/projected/826e2da1-89ab-4043-b141-3a2c9798379e-kube-api-access-mlh8c\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8\" (UID: \"826e2da1-89ab-4043-b141-3a2c9798379e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"
Sep 17 18:28:21.890726 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:21.890624    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"
Sep 17 18:28:22.166861 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:22.166829    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"]
Sep 17 18:28:22.168638 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:28:22.168611    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826e2da1_89ab_4043_b141_3a2c9798379e.slice/crio-53536c2b96feaaac2031159368224da7fda5ff56dd8b568a2a070495dc72c249 WatchSource:0}: Error finding container 53536c2b96feaaac2031159368224da7fda5ff56dd8b568a2a070495dc72c249: Status 404 returned error can't find the container with id 53536c2b96feaaac2031159368224da7fda5ff56dd8b568a2a070495dc72c249
Sep 17 18:28:23.035128 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:23.035080    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8" event={"ID":"826e2da1-89ab-4043-b141-3a2c9798379e","Type":"ContainerStarted","Data":"50e0a9ba82e3928f4aa7dcac30b8963eb55bfa09f3feb8f63a869726a9712341"}
Sep 17 18:28:23.035318 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:23.035142    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8" event={"ID":"826e2da1-89ab-4043-b141-3a2c9798379e","Type":"ContainerStarted","Data":"53536c2b96feaaac2031159368224da7fda5ff56dd8b568a2a070495dc72c249"}
Sep 17 18:28:23.035318 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:23.035182    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"
Sep 17 18:28:23.055313 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:23.055263    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8" podStartSLOduration=2.055247322 podStartE2EDuration="2.055247322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:23.05201127 +0000 UTC m=+1873.352833595" watchObservedRunningTime="2026-09-17 18:28:23.055247322 +0000 UTC m=+1873.356069646"
Sep 17 18:28:30.104987 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:30.104939    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:35.882087 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:35.882045    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"]
Sep 17 18:28:35.891858 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:35.891812    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:35.893880 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:28:35.893843    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:28:35.893880 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:28:35.893880 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:28:35.893880 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3" cmd=["/bin/sleep","15"]
Sep 17 18:28:35.894177 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:35.893909    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9" message="" err=<
Sep 17 18:28:35.894177 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:28:35.894177 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:28:35.894177 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 18:28:35.894177 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:28:35.893926    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:28:35.894177 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:28:35.894177 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:28:35.894177 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9" podUID="64ef4b97-a782-41b6-82d5-511179198afd" containerName="main" containerID="cri-o://71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3"
Sep 17 18:28:35.894177 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:35.893988    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9" podUID="64ef4b97-a782-41b6-82d5-511179198afd" containerName="main" containerID="cri-o://71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3" gracePeriod=60
Sep 17 18:28:35.994567 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:35.994493    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:36.270797 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.270770    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:28:36.418032 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.417940    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-dshm\") pod \"64ef4b97-a782-41b6-82d5-511179198afd\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") "
Sep 17 18:28:36.418032 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.417986    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ef4b97-a782-41b6-82d5-511179198afd-tls-certs\") pod \"64ef4b97-a782-41b6-82d5-511179198afd\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") "
Sep 17 18:28:36.418032 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.418023    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-model-cache\") pod \"64ef4b97-a782-41b6-82d5-511179198afd\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") "
Sep 17 18:28:36.418332 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.418072    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-tmp-dir\") pod \"64ef4b97-a782-41b6-82d5-511179198afd\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") "
Sep 17 18:28:36.418332 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.418098    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-home\") pod \"64ef4b97-a782-41b6-82d5-511179198afd\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") "
Sep 17 18:28:36.418332 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.418141    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkwxk\" (UniqueName: \"kubernetes.io/projected/64ef4b97-a782-41b6-82d5-511179198afd-kube-api-access-mkwxk\") pod \"64ef4b97-a782-41b6-82d5-511179198afd\" (UID: \"64ef4b97-a782-41b6-82d5-511179198afd\") "
Sep 17 18:28:36.418563 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.418384    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-model-cache" (OuterVolumeSpecName: "model-cache") pod "64ef4b97-a782-41b6-82d5-511179198afd" (UID: "64ef4b97-a782-41b6-82d5-511179198afd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:36.418563 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.418414    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64ef4b97-a782-41b6-82d5-511179198afd" (UID: "64ef4b97-a782-41b6-82d5-511179198afd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:36.418661 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.418585    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-home" (OuterVolumeSpecName: "home") pod "64ef4b97-a782-41b6-82d5-511179198afd" (UID: "64ef4b97-a782-41b6-82d5-511179198afd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:36.420501 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.420478    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ef4b97-a782-41b6-82d5-511179198afd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64ef4b97-a782-41b6-82d5-511179198afd" (UID: "64ef4b97-a782-41b6-82d5-511179198afd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:28:36.420609 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.420503    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-dshm" (OuterVolumeSpecName: "dshm") pod "64ef4b97-a782-41b6-82d5-511179198afd" (UID: "64ef4b97-a782-41b6-82d5-511179198afd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:36.420679 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.420661    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ef4b97-a782-41b6-82d5-511179198afd-kube-api-access-mkwxk" (OuterVolumeSpecName: "kube-api-access-mkwxk") pod "64ef4b97-a782-41b6-82d5-511179198afd" (UID: "64ef4b97-a782-41b6-82d5-511179198afd"). InnerVolumeSpecName "kube-api-access-mkwxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:28:36.518941 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.518895    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:28:36.518941 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.518931    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:28:36.518941 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.518944    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkwxk\" (UniqueName: \"kubernetes.io/projected/64ef4b97-a782-41b6-82d5-511179198afd-kube-api-access-mkwxk\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:28:36.519201 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.518957    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:28:36.519201 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.518969    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ef4b97-a782-41b6-82d5-511179198afd-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:28:36.519201 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:36.518979    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ef4b97-a782-41b6-82d5-511179198afd-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:28:37.117738 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:37.117697    2574 generic.go:356] "Generic (PLEG): container finished" podID="64ef4b97-a782-41b6-82d5-511179198afd" containerID="71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3" exitCode=0
Sep 17 18:28:37.118229 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:37.117768    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"
Sep 17 18:28:37.118229 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:37.117781    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9" event={"ID":"64ef4b97-a782-41b6-82d5-511179198afd","Type":"ContainerDied","Data":"71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3"}
Sep 17 18:28:37.118229 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:37.117842    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9" event={"ID":"64ef4b97-a782-41b6-82d5-511179198afd","Type":"ContainerDied","Data":"4f60a6eebc7ea8b86ff96b394c3a600d29845675072704ba663c9d6890f19064"}
Sep 17 18:28:37.118229 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:37.117864    2574 scope.go:117] "RemoveContainer" containerID="71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3"
Sep 17 18:28:37.127839 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:37.127822    2574 scope.go:117] "RemoveContainer" containerID="71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3"
Sep 17 18:28:37.128114 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:28:37.128093    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3\": container with ID starting with 71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3 not found: ID does not exist" containerID="71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3"
Sep 17 18:28:37.128185 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:37.128122    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3"} err="failed to get container status \"71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3\": rpc error: code = NotFound desc = could not find container \"71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3\": container with ID starting with 71e0f57f24a789cf28c96b68277e12619b673f6fb79440de9c848f8c99c3dcc3 not found: ID does not exist"
Sep 17 18:28:37.140714 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:37.140692    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"]
Sep 17 18:28:37.144678 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:37.144652    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995l6rb9"]
Sep 17 18:28:37.938974 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:37.938939    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64ef4b97-a782-41b6-82d5-511179198afd" path="/var/lib/kubelet/pods/64ef4b97-a782-41b6-82d5-511179198afd/volumes"
Sep 17 18:28:43.220781 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:43.220745    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:43.685503 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:43.685460    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:49.630955 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:49.630913    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:54.042207 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:28:54.042171    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"
Sep 17 18:29:07.809283 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:07.809245    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:07.869368 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:07.869329    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:09.652474 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:09.652436    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:13.765350 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:13.765308    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:13.837818 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:13.837758    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:24.796926 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:24.796890    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:24.805142 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:24.805112    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"]
Sep 17 18:29:24.815604 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:29:24.815556    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:29:24.815604 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:29:24.815604 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:29:24.815604 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2" cmd=["/bin/sleep","15"]
Sep 17 18:29:24.815878 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:24.815611    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn" message="" err=<
Sep 17 18:29:24.815878 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:29:24.815878 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:29:24.815878 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 18:29:24.815878 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:29:24.815624    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:29:24.815878 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:29:24.815878 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:29:24.815878 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn" podUID="38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a" containerName="main" containerID="cri-o://2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2"
Sep 17 18:29:24.815878 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:24.815671    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn" podUID="38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a" containerName="main" containerID="cri-o://2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2" gracePeriod=60
Sep 17 18:29:24.851792 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:24.851761    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:25.170116 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.170087    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:29:25.328420 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.328380    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-dshm\") pod \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") "
Sep 17 18:29:25.328658 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.328436    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-home\") pod \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") "
Sep 17 18:29:25.328658 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.328502    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-tmp-dir\") pod \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") "
Sep 17 18:29:25.328658 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.328571    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-model-cache\") pod \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") "
Sep 17 18:29:25.328658 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.328605    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-tls-certs\") pod \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") "
Sep 17 18:29:25.328880 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.328688    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwscd\" (UniqueName: \"kubernetes.io/projected/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-kube-api-access-xwscd\") pod \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\" (UID: \"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a\") "
Sep 17 18:29:25.328880 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.328728    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-home" (OuterVolumeSpecName: "home") pod "38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a" (UID: "38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:25.328880 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.328846    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a" (UID: "38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:25.328880 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.328857    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a" (UID: "38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:25.329096 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.328983    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:29:25.329096 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.328997    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:29:25.329096 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.329006    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:29:25.330881 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.330856    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-dshm" (OuterVolumeSpecName: "dshm") pod "38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a" (UID: "38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:25.331009 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.330993    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a" (UID: "38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:29:25.331058 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.331013    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-kube-api-access-xwscd" (OuterVolumeSpecName: "kube-api-access-xwscd") pod "38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a" (UID: "38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a"). InnerVolumeSpecName "kube-api-access-xwscd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:25.389633 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.389589    2574 generic.go:356] "Generic (PLEG): container finished" podID="38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a" containerID="2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2" exitCode=0
Sep 17 18:29:25.389633 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.389633    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"
Sep 17 18:29:25.389849 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.389676    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn" event={"ID":"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a","Type":"ContainerDied","Data":"2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2"}
Sep 17 18:29:25.389849 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.389735    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn" event={"ID":"38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a","Type":"ContainerDied","Data":"04926c2dbd2df61c5d323d55e3cbf5a687caf0edd8bc87d652e5d3b163fafb65"}
Sep 17 18:29:25.389849 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.389764    2574 scope.go:117] "RemoveContainer" containerID="2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2"
Sep 17 18:29:25.399806 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.399785    2574 scope.go:117] "RemoveContainer" containerID="2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2"
Sep 17 18:29:25.400148 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:29:25.400125    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2\": container with ID starting with 2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2 not found: ID does not exist" containerID="2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2"
Sep 17 18:29:25.400237 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.400158    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2"} err="failed to get container status \"2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2\": rpc error: code = NotFound desc = could not find container \"2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2\": container with ID starting with 2dc6ef3d28d6e4e488146a97c6b8cacb28872f62ce8c50daf0e5a9253e161cc2 not found: ID does not exist"
Sep 17 18:29:25.411168 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.411144    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"]
Sep 17 18:29:25.413194 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.413172    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-dkzxn"]
Sep 17 18:29:25.429877 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.429850    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:29:25.429945 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.429878    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwscd\" (UniqueName: \"kubernetes.io/projected/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-kube-api-access-xwscd\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:29:25.429945 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.429888    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:29:25.939180 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:25.939145    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a" path="/var/lib/kubelet/pods/38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a/volumes"
Sep 17 18:29:54.725716 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:29:54.725638    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:14.899189 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:14.899150    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.256969 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:16.256923    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:21.474753 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:21.474665    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:25.791238 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.791199    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"]
Sep 17 18:30:25.791773 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.791752    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ef4b97-a782-41b6-82d5-511179198afd" containerName="main"
Sep 17 18:30:25.791850 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.791777    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ef4b97-a782-41b6-82d5-511179198afd" containerName="main"
Sep 17 18:30:25.791850 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.791826    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a" containerName="main"
Sep 17 18:30:25.791850 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.791835    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a" containerName="main"
Sep 17 18:30:25.791996 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.791927    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ef4b97-a782-41b6-82d5-511179198afd" containerName="main"
Sep 17 18:30:25.791996 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.791944    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="38cfe3a3-4e9a-4946-a15a-fd6cdca90e6a" containerName="main"
Sep 17 18:30:25.796459 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.796434    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:25.799081 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.799053    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 18:30:25.799225 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.799110    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 18:30:25.800228 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.800203    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-cxtj7\""
Sep 17 18:30:25.800346 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.800209    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 18:30:25.805716 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.805689    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"]
Sep 17 18:30:25.822538 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.822483    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:25.956656 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.956624    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:25.956855 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.956687    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9dfh\" (UniqueName: \"kubernetes.io/projected/525089df-2191-41f0-92eb-a9d8345d56e1-kube-api-access-b9dfh\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:25.956855 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.956747    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525089df-2191-41f0-92eb-a9d8345d56e1-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:25.956855 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.956820    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:25.957043 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.956863    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:25.957043 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:25.956886    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:26.022534 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.022477    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:26.057467 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.057358    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525089df-2191-41f0-92eb-a9d8345d56e1-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:26.057467 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.057426    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:26.057467 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.057463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:26.057790 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.057492    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:26.057790 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.057559    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:26.057790 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.057629    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9dfh\" (UniqueName: \"kubernetes.io/projected/525089df-2191-41f0-92eb-a9d8345d56e1-kube-api-access-b9dfh\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:26.058034 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.058005    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:26.058034 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.058025    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:26.058170 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.058047    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:26.060079 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.060056    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:26.060275 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.060259    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525089df-2191-41f0-92eb-a9d8345d56e1-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:26.065568 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.065543    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9dfh\" (UniqueName: \"kubernetes.io/projected/525089df-2191-41f0-92eb-a9d8345d56e1-kube-api-access-b9dfh\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:26.108596 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.108549    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:26.384937 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.384906    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"]
Sep 17 18:30:26.386581 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:30:26.386536    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525089df_2191_41f0_92eb_a9d8345d56e1.slice/crio-9a4e0137f7d4ed2c56ec132bbab803f0fa363cee422a530dd127dfcbd1d19863 WatchSource:0}: Error finding container 9a4e0137f7d4ed2c56ec132bbab803f0fa363cee422a530dd127dfcbd1d19863: Status 404 returned error can't find the container with id 9a4e0137f7d4ed2c56ec132bbab803f0fa363cee422a530dd127dfcbd1d19863
Sep 17 18:30:26.726872 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.726782    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9" event={"ID":"525089df-2191-41f0-92eb-a9d8345d56e1","Type":"ContainerStarted","Data":"1a86a8221b85caf68b7a7985c1d9f153157b49e0ab60ec3442cb25b157ba2af0"}
Sep 17 18:30:26.726872 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:26.726836    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9" event={"ID":"525089df-2191-41f0-92eb-a9d8345d56e1","Type":"ContainerStarted","Data":"9a4e0137f7d4ed2c56ec132bbab803f0fa363cee422a530dd127dfcbd1d19863"}
Sep 17 18:30:27.109274 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:27.109236    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:28.801016 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:28.800980    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:28.950410 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:28.950366    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:36.109073 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:36.109034    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:36.122333 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:36.122303    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:36.140254 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:36.140196    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9" podStartSLOduration=11.140179922 podStartE2EDuration="11.140179922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:26.746810447 +0000 UTC m=+1997.047632774" watchObservedRunningTime="2026-09-17 18:30:36.140179922 +0000 UTC m=+2006.441002247"
Sep 17 18:30:36.793272 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:36.793233    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:30:48.956428 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:48.956339    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:49.074115 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:49.074077    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:52.260298 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:52.260256    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:59.376156 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:59.376114    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:59.381281 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:59.381248    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"]
Sep 17 18:30:59.442075 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:30:59.442038    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:04.042235 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:04.042188    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8" podUID="826e2da1-89ab-4043-b141-3a2c9798379e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:31:04.768711 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:04.768674    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:10.264094 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.264057    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"]
Sep 17 18:31:10.267531 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.267495    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.270030 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.270004    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 18:31:10.278672 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.278647    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"]
Sep 17 18:31:10.315443 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.315410    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:10.320647 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.320623    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.320725 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.320657    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46p5j\" (UniqueName: \"kubernetes.io/projected/1d754d93-2cee-43cb-96a3-3082bd89b85c-kube-api-access-46p5j\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.320725 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.320685    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.320797 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.320772    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.320836 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.320812    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d754d93-2cee-43cb-96a3-3082bd89b85c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.320872 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.320843    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.422004 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.421961    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.422215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.422014    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d754d93-2cee-43cb-96a3-3082bd89b85c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.422215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.422046    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.422215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.422097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.422215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.422122    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46p5j\" (UniqueName: \"kubernetes.io/projected/1d754d93-2cee-43cb-96a3-3082bd89b85c-kube-api-access-46p5j\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.422215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.422162    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.422490 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.422475    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.422597 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.422555    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.422666 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.422639    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.424629 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.424605    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.424832 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.424813    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d754d93-2cee-43cb-96a3-3082bd89b85c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.431085 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.431065    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46p5j\" (UniqueName: \"kubernetes.io/projected/1d754d93-2cee-43cb-96a3-3082bd89b85c-kube-api-access-46p5j\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.543835 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.543738    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:10.579183 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.579149    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:10.852134 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.852059    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"]
Sep 17 18:31:10.854531 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:31:10.854487    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d754d93_2cee_43cb_96a3_3082bd89b85c.slice/crio-c3c9931036b1882d0d76d83a15a6b6bbc13877d7e9b0bcfbdefa0b0d5df013c9 WatchSource:0}: Error finding container c3c9931036b1882d0d76d83a15a6b6bbc13877d7e9b0bcfbdefa0b0d5df013c9: Status 404 returned error can't find the container with id c3c9931036b1882d0d76d83a15a6b6bbc13877d7e9b0bcfbdefa0b0d5df013c9
Sep 17 18:31:10.965658 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.965628    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf" event={"ID":"1d754d93-2cee-43cb-96a3-3082bd89b85c","Type":"ContainerStarted","Data":"4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9"}
Sep 17 18:31:10.965835 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.965678    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf" event={"ID":"1d754d93-2cee-43cb-96a3-3082bd89b85c","Type":"ContainerStarted","Data":"c3c9931036b1882d0d76d83a15a6b6bbc13877d7e9b0bcfbdefa0b0d5df013c9"}
Sep 17 18:31:10.983698 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:10.983647    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf" podStartSLOduration=0.983631686 podStartE2EDuration="983.631686ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:10.981973502 +0000 UTC m=+2041.282795827" watchObservedRunningTime="2026-09-17 18:31:10.983631686 +0000 UTC m=+2041.284454009"
Sep 17 18:31:11.579814 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:11.579766    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:14.042195 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:14.042152    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8" podUID="826e2da1-89ab-4043-b141-3a2c9798379e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:31:14.402481 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:14.402386    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8" podUID="826e2da1-89ab-4043-b141-3a2c9798379e" containerName="main" containerID="cri-o://50e0a9ba82e3928f4aa7dcac30b8963eb55bfa09f3feb8f63a869726a9712341" gracePeriod=45
Sep 17 18:31:16.970657 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:16.970608    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:20.579392 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:20.579355    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:20.592073 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:20.592048    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:21.033968 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:21.033939    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:24.041995 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:24.041957    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8" podUID="826e2da1-89ab-4043-b141-3a2c9798379e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:31:24.042458 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:24.042084    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"
Sep 17 18:31:34.042066 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:34.042028    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8" podUID="826e2da1-89ab-4043-b141-3a2c9798379e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:31:44.042179 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:44.042132    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8" podUID="826e2da1-89ab-4043-b141-3a2c9798379e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:31:44.677052 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:44.676977    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"
Sep 17 18:31:44.683551 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:44.683524    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:44.683971 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:44.683949    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"]
Sep 17 18:31:44.694937 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:31:44.694873    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:31:44.694937 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:31:44.694937 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:31:44.694937 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9" cmd=["/bin/sleep","15"]
Sep 17 18:31:44.695282 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:44.694960    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf" message="" err=<
Sep 17 18:31:44.695282 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:31:44.695282 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:31:44.695282 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 18:31:44.695282 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:31:44.694977    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:31:44.695282 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:31:44.695282 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:31:44.695282 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf" podUID="1d754d93-2cee-43cb-96a3-3082bd89b85c" containerName="main" containerID="cri-o://4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9"
Sep 17 18:31:44.695282 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:44.695023    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf" podUID="1d754d93-2cee-43cb-96a3-3082bd89b85c" containerName="main" containerID="cri-o://4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9" gracePeriod=60
Sep 17 18:31:44.780249 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:44.780199    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:44.820997 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:44.820972    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlh8c\" (UniqueName: \"kubernetes.io/projected/826e2da1-89ab-4043-b141-3a2c9798379e-kube-api-access-mlh8c\") pod \"826e2da1-89ab-4043-b141-3a2c9798379e\" (UID: \"826e2da1-89ab-4043-b141-3a2c9798379e\") "
Sep 17 18:31:44.821122 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:44.821051    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826e2da1-89ab-4043-b141-3a2c9798379e-tls-certs\") pod \"826e2da1-89ab-4043-b141-3a2c9798379e\" (UID: \"826e2da1-89ab-4043-b141-3a2c9798379e\") "
Sep 17 18:31:44.823693 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:44.823655    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826e2da1-89ab-4043-b141-3a2c9798379e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "826e2da1-89ab-4043-b141-3a2c9798379e" (UID: "826e2da1-89ab-4043-b141-3a2c9798379e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:44.823803 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:44.823708    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826e2da1-89ab-4043-b141-3a2c9798379e-kube-api-access-mlh8c" (OuterVolumeSpecName: "kube-api-access-mlh8c") pod "826e2da1-89ab-4043-b141-3a2c9798379e" (UID: "826e2da1-89ab-4043-b141-3a2c9798379e"). InnerVolumeSpecName "kube-api-access-mlh8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:44.922361 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:44.922326    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlh8c\" (UniqueName: \"kubernetes.io/projected/826e2da1-89ab-4043-b141-3a2c9798379e-kube-api-access-mlh8c\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:31:44.922361 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:44.922354    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826e2da1-89ab-4043-b141-3a2c9798379e-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:31:44.936100 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:44.936075    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:45.123274 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.123238    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-dshm\") pod \"1d754d93-2cee-43cb-96a3-3082bd89b85c\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") "
Sep 17 18:31:45.123746 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.123294    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46p5j\" (UniqueName: \"kubernetes.io/projected/1d754d93-2cee-43cb-96a3-3082bd89b85c-kube-api-access-46p5j\") pod \"1d754d93-2cee-43cb-96a3-3082bd89b85c\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") "
Sep 17 18:31:45.123746 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.123327    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-tmp-dir\") pod \"1d754d93-2cee-43cb-96a3-3082bd89b85c\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") "
Sep 17 18:31:45.123746 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.123381    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d754d93-2cee-43cb-96a3-3082bd89b85c-tls-certs\") pod \"1d754d93-2cee-43cb-96a3-3082bd89b85c\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") "
Sep 17 18:31:45.123746 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.123422    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-model-cache\") pod \"1d754d93-2cee-43cb-96a3-3082bd89b85c\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") "
Sep 17 18:31:45.123746 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.123451    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-home\") pod \"1d754d93-2cee-43cb-96a3-3082bd89b85c\" (UID: \"1d754d93-2cee-43cb-96a3-3082bd89b85c\") "
Sep 17 18:31:45.123746 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.123662    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d754d93-2cee-43cb-96a3-3082bd89b85c" (UID: "1d754d93-2cee-43cb-96a3-3082bd89b85c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:45.124004 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.123768    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d754d93-2cee-43cb-96a3-3082bd89b85c" (UID: "1d754d93-2cee-43cb-96a3-3082bd89b85c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:45.124004 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.123782    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-home" (OuterVolumeSpecName: "home") pod "1d754d93-2cee-43cb-96a3-3082bd89b85c" (UID: "1d754d93-2cee-43cb-96a3-3082bd89b85c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:45.125906 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.125881    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d754d93-2cee-43cb-96a3-3082bd89b85c-kube-api-access-46p5j" (OuterVolumeSpecName: "kube-api-access-46p5j") pod "1d754d93-2cee-43cb-96a3-3082bd89b85c" (UID: "1d754d93-2cee-43cb-96a3-3082bd89b85c"). InnerVolumeSpecName "kube-api-access-46p5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:45.125906 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.125895    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d754d93-2cee-43cb-96a3-3082bd89b85c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d754d93-2cee-43cb-96a3-3082bd89b85c" (UID: "1d754d93-2cee-43cb-96a3-3082bd89b85c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:45.126013 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.125908    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-dshm" (OuterVolumeSpecName: "dshm") pod "1d754d93-2cee-43cb-96a3-3082bd89b85c" (UID: "1d754d93-2cee-43cb-96a3-3082bd89b85c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:45.154435 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.154398    2574 generic.go:356] "Generic (PLEG): container finished" podID="1d754d93-2cee-43cb-96a3-3082bd89b85c" containerID="4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9" exitCode=0
Sep 17 18:31:45.154653 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.154448    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"
Sep 17 18:31:45.154653 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.154480    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf" event={"ID":"1d754d93-2cee-43cb-96a3-3082bd89b85c","Type":"ContainerDied","Data":"4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9"}
Sep 17 18:31:45.154653 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.154548    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf" event={"ID":"1d754d93-2cee-43cb-96a3-3082bd89b85c","Type":"ContainerDied","Data":"c3c9931036b1882d0d76d83a15a6b6bbc13877d7e9b0bcfbdefa0b0d5df013c9"}
Sep 17 18:31:45.154653 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.154573    2574 scope.go:117] "RemoveContainer" containerID="4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9"
Sep 17 18:31:45.155883 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.155853    2574 generic.go:356] "Generic (PLEG): container finished" podID="826e2da1-89ab-4043-b141-3a2c9798379e" containerID="50e0a9ba82e3928f4aa7dcac30b8963eb55bfa09f3feb8f63a869726a9712341" exitCode=0
Sep 17 18:31:45.156003 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.155906    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"
Sep 17 18:31:45.156003 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.155932    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8" event={"ID":"826e2da1-89ab-4043-b141-3a2c9798379e","Type":"ContainerDied","Data":"50e0a9ba82e3928f4aa7dcac30b8963eb55bfa09f3feb8f63a869726a9712341"}
Sep 17 18:31:45.156003 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.155967    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8" event={"ID":"826e2da1-89ab-4043-b141-3a2c9798379e","Type":"ContainerDied","Data":"53536c2b96feaaac2031159368224da7fda5ff56dd8b568a2a070495dc72c249"}
Sep 17 18:31:45.164991 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.164971    2574 scope.go:117] "RemoveContainer" containerID="4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9"
Sep 17 18:31:45.165298 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:31:45.165274    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9\": container with ID starting with 4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9 not found: ID does not exist" containerID="4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9"
Sep 17 18:31:45.165381 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.165312    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9"} err="failed to get container status \"4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9\": rpc error: code = NotFound desc = could not find container \"4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9\": container with ID starting with 4dee1c0ee9525632e8125630024e2318668c6b367de7aee82c9fcf67fa6d1cf9 not found: ID does not exist"
Sep 17 18:31:45.165381 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.165332    2574 scope.go:117] "RemoveContainer" containerID="50e0a9ba82e3928f4aa7dcac30b8963eb55bfa09f3feb8f63a869726a9712341"
Sep 17 18:31:45.174098 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.174075    2574 scope.go:117] "RemoveContainer" containerID="50e0a9ba82e3928f4aa7dcac30b8963eb55bfa09f3feb8f63a869726a9712341"
Sep 17 18:31:45.174395 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:31:45.174374    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e0a9ba82e3928f4aa7dcac30b8963eb55bfa09f3feb8f63a869726a9712341\": container with ID starting with 50e0a9ba82e3928f4aa7dcac30b8963eb55bfa09f3feb8f63a869726a9712341 not found: ID does not exist" containerID="50e0a9ba82e3928f4aa7dcac30b8963eb55bfa09f3feb8f63a869726a9712341"
Sep 17 18:31:45.174468 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.174400    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e0a9ba82e3928f4aa7dcac30b8963eb55bfa09f3feb8f63a869726a9712341"} err="failed to get container status \"50e0a9ba82e3928f4aa7dcac30b8963eb55bfa09f3feb8f63a869726a9712341\": rpc error: code = NotFound desc = could not find container \"50e0a9ba82e3928f4aa7dcac30b8963eb55bfa09f3feb8f63a869726a9712341\": container with ID starting with 50e0a9ba82e3928f4aa7dcac30b8963eb55bfa09f3feb8f63a869726a9712341 not found: ID does not exist"
Sep 17 18:31:45.176446 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.176425    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"]
Sep 17 18:31:45.182163 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.182102    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-79qpf"]
Sep 17 18:31:45.192587 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.192562    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"]
Sep 17 18:31:45.198383 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.198352    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche94sc8"]
Sep 17 18:31:45.224446 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.224417    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d754d93-2cee-43cb-96a3-3082bd89b85c-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:31:45.224446 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.224445    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:31:45.224643 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.224454    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:31:45.224643 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.224461    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:31:45.224643 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.224469    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46p5j\" (UniqueName: \"kubernetes.io/projected/1d754d93-2cee-43cb-96a3-3082bd89b85c-kube-api-access-46p5j\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:31:45.224643 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.224499    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d754d93-2cee-43cb-96a3-3082bd89b85c-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:31:45.938188 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.938152    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d754d93-2cee-43cb-96a3-3082bd89b85c" path="/var/lib/kubelet/pods/1d754d93-2cee-43cb-96a3-3082bd89b85c/volumes"
Sep 17 18:31:45.938553 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:45.938540    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="826e2da1-89ab-4043-b141-3a2c9798379e" path="/var/lib/kubelet/pods/826e2da1-89ab-4043-b141-3a2c9798379e/volumes"
Sep 17 18:31:48.221772 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:48.221728    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:48.320057 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:48.320019    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:54.719438 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:31:54.719387    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:00.580155 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:00.580119    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:00.656403 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:00.656370    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:09.049892 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.049856    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"]
Sep 17 18:32:09.050414 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.050395    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d754d93-2cee-43cb-96a3-3082bd89b85c" containerName="main"
Sep 17 18:32:09.050485 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.050417    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d754d93-2cee-43cb-96a3-3082bd89b85c" containerName="main"
Sep 17 18:32:09.050485 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.050436    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="826e2da1-89ab-4043-b141-3a2c9798379e" containerName="main"
Sep 17 18:32:09.050485 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.050441    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="826e2da1-89ab-4043-b141-3a2c9798379e" containerName="main"
Sep 17 18:32:09.050638 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.050499    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d754d93-2cee-43cb-96a3-3082bd89b85c" containerName="main"
Sep 17 18:32:09.050638 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.050539    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="826e2da1-89ab-4043-b141-3a2c9798379e" containerName="main"
Sep 17 18:32:09.053783 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.053762    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.056319 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.056291    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 18:32:09.056457 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.056336    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-fr6c6\""
Sep 17 18:32:09.056457 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.056413    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 18:32:09.056621 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.056472    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 18:32:09.065018 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.064990    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"]
Sep 17 18:32:09.093596 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.093562    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:09.113679 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.113646    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.113861 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.113702    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.113861 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.113730    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.113861 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.113763    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.113861 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.113791    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.113861 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.113839    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b326a608-3a96-4091-954e-c6dd6d37a17c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.114041 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.113872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfgwb\" (UniqueName: \"kubernetes.io/projected/b326a608-3a96-4091-954e-c6dd6d37a17c-kube-api-access-mfgwb\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.215026 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.214987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfgwb\" (UniqueName: \"kubernetes.io/projected/b326a608-3a96-4091-954e-c6dd6d37a17c-kube-api-access-mfgwb\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.215216 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.215035    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.215216 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.215080    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.215216 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.215097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.215216 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.215133    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.215216 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.215160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.215449 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.215324    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b326a608-3a96-4091-954e-c6dd6d37a17c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.215625 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.215604    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.215710 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.215603    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.215710 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.215656    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.215710 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.215684    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.217818 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.217793    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.218060 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.218042    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b326a608-3a96-4091-954e-c6dd6d37a17c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.223122 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.223090    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfgwb\" (UniqueName: \"kubernetes.io/projected/b326a608-3a96-4091-954e-c6dd6d37a17c-kube-api-access-mfgwb\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.357766 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.357673    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:09.365675 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.365651    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:09.644615 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.644585    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"]
Sep 17 18:32:09.691954 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:09.691935    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:32:10.297915 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:10.297882    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" event={"ID":"b326a608-3a96-4091-954e-c6dd6d37a17c","Type":"ContainerStarted","Data":"7bac75451e19a7e67452e535e9de01080c0908bc3ec3ed2e59babb4dd67cb7f1"}
Sep 17 18:32:11.977667 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:11.977626    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:32:11.986940 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:11.986919    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:32:17.343097 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:17.343065    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" event={"ID":"b326a608-3a96-4091-954e-c6dd6d37a17c","Type":"ContainerStarted","Data":"72174bbb853c6d15d18591eb2c87b9ff0c32392a00f3ef995b7e4d8534ba424b"}
Sep 17 18:32:19.354807 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:19.354770    2574 generic.go:356] "Generic (PLEG): container finished" podID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerID="72174bbb853c6d15d18591eb2c87b9ff0c32392a00f3ef995b7e4d8534ba424b" exitCode=0
Sep 17 18:32:19.355216 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:19.354839    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" event={"ID":"b326a608-3a96-4091-954e-c6dd6d37a17c","Type":"ContainerDied","Data":"72174bbb853c6d15d18591eb2c87b9ff0c32392a00f3ef995b7e4d8534ba424b"}
Sep 17 18:32:20.364155 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:20.364122    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" event={"ID":"b326a608-3a96-4091-954e-c6dd6d37a17c","Type":"ContainerStarted","Data":"281343cca218b2716736123955ba493c679aaf02e27ae571f9f1b245a4924ea3"}
Sep 17 18:32:20.364535 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:20.364194    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" event={"ID":"b326a608-3a96-4091-954e-c6dd6d37a17c","Type":"ContainerStarted","Data":"8f7eb95add7a63fb08af3b135fe13716c82fcb288874d0cc99cb9d824670e145"}
Sep 17 18:32:20.366583 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:20.366564    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:20.386730 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:20.386681    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" podStartSLOduration=2.417014061 podStartE2EDuration="11.386667889s" totalImagesPullingTime="7.784384803s" totalInitContainerRuntime="1.185269025s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:20.384738457 +0000 UTC m=+2110.685560805" watchObservedRunningTime="2026-09-17 18:32:20.386667889 +0000 UTC m=+2110.687490213"
Sep 17 18:32:29.366582 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:29.366544    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:32:29.368181 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:29.368145    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 18:32:39.355447 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:39.355362    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:39.366086 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:39.366049    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 18:32:40.924473 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:40.924434    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:40.987462 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:40.987425    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"]
Sep 17 18:32:40.996740 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:32:40.996692    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:32:40.996740 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:32:40.996740 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:32:40.996740 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="1a86a8221b85caf68b7a7985c1d9f153157b49e0ab60ec3442cb25b157ba2af0" cmd=["/bin/sleep","15"]
Sep 17 18:32:40.997004 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:40.996752    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9" message="" err=<
Sep 17 18:32:40.997004 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:32:40.997004 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:32:40.997004 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 18:32:40.997004 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:32:40.996776    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:32:40.997004 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:32:40.997004 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 18:32:40.997004 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9" podUID="525089df-2191-41f0-92eb-a9d8345d56e1" containerName="main" containerID="cri-o://1a86a8221b85caf68b7a7985c1d9f153157b49e0ab60ec3442cb25b157ba2af0"
Sep 17 18:32:40.997004 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:40.996815    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9" podUID="525089df-2191-41f0-92eb-a9d8345d56e1" containerName="main" containerID="cri-o://1a86a8221b85caf68b7a7985c1d9f153157b49e0ab60ec3442cb25b157ba2af0" gracePeriod=60
Sep 17 18:32:41.052682 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.052642    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:41.489696 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.489663    2574 generic.go:356] "Generic (PLEG): container finished" podID="525089df-2191-41f0-92eb-a9d8345d56e1" containerID="1a86a8221b85caf68b7a7985c1d9f153157b49e0ab60ec3442cb25b157ba2af0" exitCode=0
Sep 17 18:32:41.489871 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.489725    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9" event={"ID":"525089df-2191-41f0-92eb-a9d8345d56e1","Type":"ContainerDied","Data":"1a86a8221b85caf68b7a7985c1d9f153157b49e0ab60ec3442cb25b157ba2af0"}
Sep 17 18:32:41.767228 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.767200    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:32:41.807056 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.807017    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-tmp-dir\") pod \"525089df-2191-41f0-92eb-a9d8345d56e1\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") "
Sep 17 18:32:41.807248 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.807084    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-model-cache\") pod \"525089df-2191-41f0-92eb-a9d8345d56e1\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") "
Sep 17 18:32:41.807248 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.807130    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9dfh\" (UniqueName: \"kubernetes.io/projected/525089df-2191-41f0-92eb-a9d8345d56e1-kube-api-access-b9dfh\") pod \"525089df-2191-41f0-92eb-a9d8345d56e1\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") "
Sep 17 18:32:41.807248 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.807156    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525089df-2191-41f0-92eb-a9d8345d56e1-tls-certs\") pod \"525089df-2191-41f0-92eb-a9d8345d56e1\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") "
Sep 17 18:32:41.807248 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.807181    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-dshm\") pod \"525089df-2191-41f0-92eb-a9d8345d56e1\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") "
Sep 17 18:32:41.807248 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.807216    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-home\") pod \"525089df-2191-41f0-92eb-a9d8345d56e1\" (UID: \"525089df-2191-41f0-92eb-a9d8345d56e1\") "
Sep 17 18:32:41.807543 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.807321    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "525089df-2191-41f0-92eb-a9d8345d56e1" (UID: "525089df-2191-41f0-92eb-a9d8345d56e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:41.807608 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.807490    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:32:41.807608 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.807581    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-home" (OuterVolumeSpecName: "home") pod "525089df-2191-41f0-92eb-a9d8345d56e1" (UID: "525089df-2191-41f0-92eb-a9d8345d56e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:41.807759 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.807729    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "525089df-2191-41f0-92eb-a9d8345d56e1" (UID: "525089df-2191-41f0-92eb-a9d8345d56e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:41.809639 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.809609    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-dshm" (OuterVolumeSpecName: "dshm") pod "525089df-2191-41f0-92eb-a9d8345d56e1" (UID: "525089df-2191-41f0-92eb-a9d8345d56e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:41.810083 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.810063    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525089df-2191-41f0-92eb-a9d8345d56e1-kube-api-access-b9dfh" (OuterVolumeSpecName: "kube-api-access-b9dfh") pod "525089df-2191-41f0-92eb-a9d8345d56e1" (UID: "525089df-2191-41f0-92eb-a9d8345d56e1"). InnerVolumeSpecName "kube-api-access-b9dfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:41.810365 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.810347    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525089df-2191-41f0-92eb-a9d8345d56e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "525089df-2191-41f0-92eb-a9d8345d56e1" (UID: "525089df-2191-41f0-92eb-a9d8345d56e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:41.907934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.907899    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:32:41.907934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.907928    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9dfh\" (UniqueName: \"kubernetes.io/projected/525089df-2191-41f0-92eb-a9d8345d56e1-kube-api-access-b9dfh\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:32:41.907934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.907937    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525089df-2191-41f0-92eb-a9d8345d56e1-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:32:41.908171 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.907947    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:32:41.908171 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:41.907957    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525089df-2191-41f0-92eb-a9d8345d56e1-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:32:42.497192 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:42.497158    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9" event={"ID":"525089df-2191-41f0-92eb-a9d8345d56e1","Type":"ContainerDied","Data":"9a4e0137f7d4ed2c56ec132bbab803f0fa363cee422a530dd127dfcbd1d19863"}
Sep 17 18:32:42.497663 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:42.497244    2574 scope.go:117] "RemoveContainer" containerID="1a86a8221b85caf68b7a7985c1d9f153157b49e0ab60ec3442cb25b157ba2af0"
Sep 17 18:32:42.497663 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:42.497167    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"
Sep 17 18:32:42.516793 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:42.516765    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"]
Sep 17 18:32:42.520276 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:42.520253    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969nskd9"]
Sep 17 18:32:43.939328 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:43.939295    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="525089df-2191-41f0-92eb-a9d8345d56e1" path="/var/lib/kubelet/pods/525089df-2191-41f0-92eb-a9d8345d56e1/volumes"
Sep 17 18:32:49.366663 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:49.366621    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 18:32:59.366544 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:32:59.366434    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 18:33:09.366668 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:33:09.366629    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 18:33:19.366731 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:33:19.366674    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 18:33:29.366813 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:33:29.366769    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 18:33:29.565818 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:33:29.565782    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:29.658619 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:33:29.658473    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:36.318721 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:33:36.318686    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:39.366457 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:33:39.366415    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 18:33:42.298590 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:33:42.298558    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:42.399946 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:33:42.399909    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:44.629805 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:33:44.629751    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:44.802797 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:33:44.802765    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:49.366566 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:33:49.366491    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 18:33:53.805465 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:33:53.805422    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:59.366706 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:33:59.366658    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 18:34:09.366298 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:09.366238    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 18:34:19.377152 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:19.377117    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:34:19.385682 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:19.385648    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:34:24.424250 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:24.424163    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"]
Sep 17 18:34:24.424776 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:24.424634    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="modelcar" containerID="cri-o://281343cca218b2716736123955ba493c679aaf02e27ae571f9f1b245a4924ea3" gracePeriod=60
Sep 17 18:34:24.437287 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:24.437240    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:24.538457 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:24.538426    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:25.102869 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:25.102835    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56_b326a608-3a96-4091-954e-c6dd6d37a17c/modelcar/0.log"
Sep 17 18:34:25.103670 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:25.103639    2574 generic.go:356] "Generic (PLEG): container finished" podID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerID="281343cca218b2716736123955ba493c679aaf02e27ae571f9f1b245a4924ea3" exitCode=143
Sep 17 18:34:25.103807 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:25.103696    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" event={"ID":"b326a608-3a96-4091-954e-c6dd6d37a17c","Type":"ContainerDied","Data":"281343cca218b2716736123955ba493c679aaf02e27ae571f9f1b245a4924ea3"}
Sep 17 18:34:39.448987 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:39.448937    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main" containerID="cri-o://8f7eb95add7a63fb08af3b135fe13716c82fcb288874d0cc99cb9d824670e145" gracePeriod=45
Sep 17 18:34:40.186050 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.186020    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56_b326a608-3a96-4091-954e-c6dd6d37a17c/modelcar/0.log"
Sep 17 18:34:40.186838 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.186807    2574 generic.go:356] "Generic (PLEG): container finished" podID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerID="8f7eb95add7a63fb08af3b135fe13716c82fcb288874d0cc99cb9d824670e145" exitCode=0
Sep 17 18:34:40.186963 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.186878    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" event={"ID":"b326a608-3a96-4091-954e-c6dd6d37a17c","Type":"ContainerDied","Data":"8f7eb95add7a63fb08af3b135fe13716c82fcb288874d0cc99cb9d824670e145"}
Sep 17 18:34:40.337943 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.337918    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56_b326a608-3a96-4091-954e-c6dd6d37a17c/modelcar/0.log"
Sep 17 18:34:40.340734 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.340716    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:34:40.422711 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.422668    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfgwb\" (UniqueName: \"kubernetes.io/projected/b326a608-3a96-4091-954e-c6dd6d37a17c-kube-api-access-mfgwb\") pod \"b326a608-3a96-4091-954e-c6dd6d37a17c\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") "
Sep 17 18:34:40.422915 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.422732    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-dshm\") pod \"b326a608-3a96-4091-954e-c6dd6d37a17c\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") "
Sep 17 18:34:40.422915 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.422796    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-home\") pod \"b326a608-3a96-4091-954e-c6dd6d37a17c\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") "
Sep 17 18:34:40.422915 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.422821    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-model-cache\") pod \"b326a608-3a96-4091-954e-c6dd6d37a17c\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") "
Sep 17 18:34:40.422915 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.422879    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-kserve-provision-location\") pod \"b326a608-3a96-4091-954e-c6dd6d37a17c\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") "
Sep 17 18:34:40.423191 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.422931    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-tmp-dir\") pod \"b326a608-3a96-4091-954e-c6dd6d37a17c\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") "
Sep 17 18:34:40.423191 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.422961    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b326a608-3a96-4091-954e-c6dd6d37a17c-tls-certs\") pod \"b326a608-3a96-4091-954e-c6dd6d37a17c\" (UID: \"b326a608-3a96-4091-954e-c6dd6d37a17c\") "
Sep 17 18:34:40.423191 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.423112    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b326a608-3a96-4091-954e-c6dd6d37a17c" (UID: "b326a608-3a96-4091-954e-c6dd6d37a17c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:40.423191 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.423168    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b326a608-3a96-4091-954e-c6dd6d37a17c" (UID: "b326a608-3a96-4091-954e-c6dd6d37a17c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:40.423369 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.423265    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:34:40.423369 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.423284    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:34:40.423760 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.423735    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-home" (OuterVolumeSpecName: "home") pod "b326a608-3a96-4091-954e-c6dd6d37a17c" (UID: "b326a608-3a96-4091-954e-c6dd6d37a17c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:40.425899 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.425866    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-dshm" (OuterVolumeSpecName: "dshm") pod "b326a608-3a96-4091-954e-c6dd6d37a17c" (UID: "b326a608-3a96-4091-954e-c6dd6d37a17c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:40.425899 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.425887    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b326a608-3a96-4091-954e-c6dd6d37a17c-kube-api-access-mfgwb" (OuterVolumeSpecName: "kube-api-access-mfgwb") pod "b326a608-3a96-4091-954e-c6dd6d37a17c" (UID: "b326a608-3a96-4091-954e-c6dd6d37a17c"). InnerVolumeSpecName "kube-api-access-mfgwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:34:40.426038 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.425930    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b326a608-3a96-4091-954e-c6dd6d37a17c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b326a608-3a96-4091-954e-c6dd6d37a17c" (UID: "b326a608-3a96-4091-954e-c6dd6d37a17c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:34:40.436371 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.436340    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b326a608-3a96-4091-954e-c6dd6d37a17c" (UID: "b326a608-3a96-4091-954e-c6dd6d37a17c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:40.524018 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.523984    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:34:40.524018 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.524012    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b326a608-3a96-4091-954e-c6dd6d37a17c-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:34:40.524018 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.524021    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfgwb\" (UniqueName: \"kubernetes.io/projected/b326a608-3a96-4091-954e-c6dd6d37a17c-kube-api-access-mfgwb\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:34:40.524481 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.524031    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:34:40.524481 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:40.524039    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b326a608-3a96-4091-954e-c6dd6d37a17c-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:34:41.194030 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:41.194000    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56_b326a608-3a96-4091-954e-c6dd6d37a17c/modelcar/0.log"
Sep 17 18:34:41.194772 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:41.194753    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"
Sep 17 18:34:41.194896 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:41.194749    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56" event={"ID":"b326a608-3a96-4091-954e-c6dd6d37a17c","Type":"ContainerDied","Data":"7bac75451e19a7e67452e535e9de01080c0908bc3ec3ed2e59babb4dd67cb7f1"}
Sep 17 18:34:41.194937 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:41.194894    2574 scope.go:117] "RemoveContainer" containerID="281343cca218b2716736123955ba493c679aaf02e27ae571f9f1b245a4924ea3"
Sep 17 18:34:41.204659 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:41.204633    2574 scope.go:117] "RemoveContainer" containerID="8f7eb95add7a63fb08af3b135fe13716c82fcb288874d0cc99cb9d824670e145"
Sep 17 18:34:41.213278 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:41.213257    2574 scope.go:117] "RemoveContainer" containerID="72174bbb853c6d15d18591eb2c87b9ff0c32392a00f3ef995b7e4d8534ba424b"
Sep 17 18:34:41.217545 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:41.217502    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"]
Sep 17 18:34:41.221215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:41.221189    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgqq56"]
Sep 17 18:34:41.938410 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:34:41.938374    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" path="/var/lib/kubelet/pods/b326a608-3a96-4091-954e-c6dd6d37a17c/volumes"
Sep 17 18:35:05.649862 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:05.649827    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:14.541647 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:14.541606    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:14.651416 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:14.651366    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:21.102825 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:21.102780    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:26.887624 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:26.887588    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:26.950954 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:26.950913    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:43.881998 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.881958    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"]
Sep 17 18:35:43.882408 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.882362    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="modelcar"
Sep 17 18:35:43.882408 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.882374    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="modelcar"
Sep 17 18:35:43.882408 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.882393    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="modelcar-init"
Sep 17 18:35:43.882408 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.882398    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="modelcar-init"
Sep 17 18:35:43.882408 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.882403    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main"
Sep 17 18:35:43.882408 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.882411    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main"
Sep 17 18:35:43.882629 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.882421    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="525089df-2191-41f0-92eb-a9d8345d56e1" containerName="main"
Sep 17 18:35:43.882629 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.882427    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="525089df-2191-41f0-92eb-a9d8345d56e1" containerName="main"
Sep 17 18:35:43.882629 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.882475    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="modelcar"
Sep 17 18:35:43.882629 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.882485    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b326a608-3a96-4091-954e-c6dd6d37a17c" containerName="main"
Sep 17 18:35:43.882629 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.882491    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="525089df-2191-41f0-92eb-a9d8345d56e1" containerName="main"
Sep 17 18:35:43.885745 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.885725    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:43.888093 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.888066    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:35:43.888218 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.888197    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 18:35:43.888286 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.888201    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 18:35:43.888506 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.888490    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 18:35:43.888725 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.888702    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-p6c9l\""
Sep 17 18:35:43.888834 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.888733    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 17 18:35:43.896058 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.896032    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"]
Sep 17 18:35:43.980379 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.980341    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkcf5\" (UniqueName: \"kubernetes.io/projected/a5acb7e3-69e6-4452-874a-d2b2041cc643-kube-api-access-qkcf5\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:43.980379 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.980383    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5acb7e3-69e6-4452-874a-d2b2041cc643-cabundle-cert\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:43.980632 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.980405    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:43.980632 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.980475    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5acb7e3-69e6-4452-874a-d2b2041cc643-tls-certs\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:43.980632 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.980548    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-model-cache\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:43.980632 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.980599    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-dshm\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:43.980632 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.980616    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-tmp-dir\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:43.980818 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:43.980645    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-home\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.081598 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.081560    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkcf5\" (UniqueName: \"kubernetes.io/projected/a5acb7e3-69e6-4452-874a-d2b2041cc643-kube-api-access-qkcf5\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.081819 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.081612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5acb7e3-69e6-4452-874a-d2b2041cc643-cabundle-cert\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.081819 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.081643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.081819 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.081674    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5acb7e3-69e6-4452-874a-d2b2041cc643-tls-certs\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.081819 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.081710    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-model-cache\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.081819 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.081778    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-dshm\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.081819 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.081799    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-tmp-dir\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.082140 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.081842    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-home\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.082205 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.082146    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.082205 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.082187    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-home\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.082506 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.082443    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-model-cache\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.082506 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.082497    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-tmp-dir\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.083153 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.083123    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5acb7e3-69e6-4452-874a-d2b2041cc643-cabundle-cert\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.084636 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.084609    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:44.085121 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.085099    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-dshm\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.085493 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.085472    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5acb7e3-69e6-4452-874a-d2b2041cc643-tls-certs\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.090218 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.090197    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkcf5\" (UniqueName: \"kubernetes.io/projected/a5acb7e3-69e6-4452-874a-d2b2041cc643-kube-api-access-qkcf5\") pod \"custom-route-timeout-test-kserve-76dbdf99f6-hdnsh\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.197911 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.197803    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:44.355928 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.355880    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:44.477717 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.477688    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"]
Sep 17 18:35:44.479415 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:35:44.479382    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5acb7e3_69e6_4452_874a_d2b2041cc643.slice/crio-790a30deed972c77297af31088abc5473d90e23313ada3f3476b2120c7b29715 WatchSource:0}: Error finding container 790a30deed972c77297af31088abc5473d90e23313ada3f3476b2120c7b29715: Status 404 returned error can't find the container with id 790a30deed972c77297af31088abc5473d90e23313ada3f3476b2120c7b29715
Sep 17 18:35:44.530301 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:44.530262    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" event={"ID":"a5acb7e3-69e6-4452-874a-d2b2041cc643","Type":"ContainerStarted","Data":"790a30deed972c77297af31088abc5473d90e23313ada3f3476b2120c7b29715"}
Sep 17 18:35:45.536972 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:45.536922    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" event={"ID":"a5acb7e3-69e6-4452-874a-d2b2041cc643","Type":"ContainerStarted","Data":"d941d1f3eb42b05646fb2f3def887d475cc1fd0501a676b134a210b27d589fc1"}
Sep 17 18:35:47.549909 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:47.549871    2574 generic.go:356] "Generic (PLEG): container finished" podID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerID="d941d1f3eb42b05646fb2f3def887d475cc1fd0501a676b134a210b27d589fc1" exitCode=0
Sep 17 18:35:47.550320 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:47.549946    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" event={"ID":"a5acb7e3-69e6-4452-874a-d2b2041cc643","Type":"ContainerDied","Data":"d941d1f3eb42b05646fb2f3def887d475cc1fd0501a676b134a210b27d589fc1"}
Sep 17 18:35:48.558428 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:48.558383    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" event={"ID":"a5acb7e3-69e6-4452-874a-d2b2041cc643","Type":"ContainerStarted","Data":"2f0d0d3fb02447f9364f55ead14f8a5aa8d915d15afaad9eda19e5a70f78961b"}
Sep 17 18:35:48.580588 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:48.580506    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" podStartSLOduration=3.330243999 podStartE2EDuration="5.580490346s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.250246347s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:48.578173561 +0000 UTC m=+2318.878995897" watchObservedRunningTime="2026-09-17 18:35:48.580490346 +0000 UTC m=+2318.881312669"
Sep 17 18:35:49.198840 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:49.198803    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:54.198459 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:54.198357    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:35:54.200078 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:54.200043    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:35:58.637455 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:35:58.637411    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:00.197853 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:00.197818    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:00.260411 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:00.260374    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:04.199243 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:04.199190    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:36:14.198576 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:14.198531    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:36:23.171921 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:23.171878    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:24.198829 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:24.198784    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:36:34.198957 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:34.198902    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:36:44.198619 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:44.198567    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:36:46.379650 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:46.379613    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:46.485434 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:46.485400    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:46.582733 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:46.582693    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:52.936307 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:52.936268    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:53.039954 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:53.039917    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:54.198583 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:54.198509    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:36:58.873969 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:58.873920    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:58.946751 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:36:58.946708    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:04.198800 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:04.198748    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:37:05.214243 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.214174    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:05.436232 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.436196    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"]
Sep 17 18:37:05.439255 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.439236    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"
Sep 17 18:37:05.442658 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.442620    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 18:37:05.442658 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.442622    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-7gmpp\""
Sep 17 18:37:05.442907 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.442620    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 18:37:05.442907 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.442620    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-qnzgf\""
Sep 17 18:37:05.442907 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.442627    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 18:37:05.447995 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.447969    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"]
Sep 17 18:37:05.464754 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.464675    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:05.516998 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.516959    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92\" (UID: \"8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"
Sep 17 18:37:05.517175 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.517052    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qngdz\" (UniqueName: \"kubernetes.io/projected/8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e-kube-api-access-qngdz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92\" (UID: \"8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"
Sep 17 18:37:05.618318 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.618284    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92\" (UID: \"8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"
Sep 17 18:37:05.618530 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.618352    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngdz\" (UniqueName: \"kubernetes.io/projected/8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e-kube-api-access-qngdz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92\" (UID: \"8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"
Sep 17 18:37:05.621112 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.621082    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92\" (UID: \"8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"
Sep 17 18:37:05.626995 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.626964    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qngdz\" (UniqueName: \"kubernetes.io/projected/8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e-kube-api-access-qngdz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92\" (UID: \"8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"
Sep 17 18:37:05.752954 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:05.752919    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"
Sep 17 18:37:06.028260 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:06.028235    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"]
Sep 17 18:37:06.030233 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:37:06.030206    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6d3e5f_9908_496a_9cd2_a6d21ab64c2e.slice/crio-5cd97795ee8b1c3d1f38e14573167869e3eb2955a6d686727d895b7aef198da8 WatchSource:0}: Error finding container 5cd97795ee8b1c3d1f38e14573167869e3eb2955a6d686727d895b7aef198da8: Status 404 returned error can't find the container with id 5cd97795ee8b1c3d1f38e14573167869e3eb2955a6d686727d895b7aef198da8
Sep 17 18:37:07.028855 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:07.028808    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92" event={"ID":"8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e","Type":"ContainerStarted","Data":"88a69d2c76de9868690f326e038feb9f921bf6e0f4f80e6465039aa9d6c986d1"}
Sep 17 18:37:07.029336 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:07.029013    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"
Sep 17 18:37:07.029336 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:07.029050    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92" event={"ID":"8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e","Type":"ContainerStarted","Data":"5cd97795ee8b1c3d1f38e14573167869e3eb2955a6d686727d895b7aef198da8"}
Sep 17 18:37:07.047997 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:07.047940    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92" podStartSLOduration=2.047924139 podStartE2EDuration="2.047924139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:07.046218655 +0000 UTC m=+2397.347040981" watchObservedRunningTime="2026-09-17 18:37:07.047924139 +0000 UTC m=+2397.348746462"
Sep 17 18:37:12.020634 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:12.020599    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:37:12.032216 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:12.032184    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:37:14.199150 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:14.199105    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:37:24.198845 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:24.198747    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:37:34.198944 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:34.198904    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:37:38.036370 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:38.036336    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"
Sep 17 18:37:41.206107 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:41.206064    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:44.208813 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:44.208781    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:37:44.217115 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:44.217077    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:37:48.797255 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:48.797215    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:48.807870 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:48.807839    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"]
Sep 17 18:37:48.851015 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:37:48.850979    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:03.832173 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:03.832130    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main" containerID="cri-o://2f0d0d3fb02447f9364f55ead14f8a5aa8d915d15afaad9eda19e5a70f78961b" gracePeriod=45
Sep 17 18:38:04.209770 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.209665    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 18:38:04.585955 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.585921    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:38:04.730257 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.730213    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-model-cache\") pod \"a5acb7e3-69e6-4452-874a-d2b2041cc643\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") "
Sep 17 18:38:04.730257 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.730259    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-dshm\") pod \"a5acb7e3-69e6-4452-874a-d2b2041cc643\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") "
Sep 17 18:38:04.730898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.730304    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-tmp-dir\") pod \"a5acb7e3-69e6-4452-874a-d2b2041cc643\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") "
Sep 17 18:38:04.730898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.730323    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5acb7e3-69e6-4452-874a-d2b2041cc643-cabundle-cert\") pod \"a5acb7e3-69e6-4452-874a-d2b2041cc643\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") "
Sep 17 18:38:04.730898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.730346    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-kserve-provision-location\") pod \"a5acb7e3-69e6-4452-874a-d2b2041cc643\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") "
Sep 17 18:38:04.730898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.730377    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-home\") pod \"a5acb7e3-69e6-4452-874a-d2b2041cc643\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") "
Sep 17 18:38:04.730898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.730407    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5acb7e3-69e6-4452-874a-d2b2041cc643-tls-certs\") pod \"a5acb7e3-69e6-4452-874a-d2b2041cc643\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") "
Sep 17 18:38:04.730898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.730444    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkcf5\" (UniqueName: \"kubernetes.io/projected/a5acb7e3-69e6-4452-874a-d2b2041cc643-kube-api-access-qkcf5\") pod \"a5acb7e3-69e6-4452-874a-d2b2041cc643\" (UID: \"a5acb7e3-69e6-4452-874a-d2b2041cc643\") "
Sep 17 18:38:04.730898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.730630    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5acb7e3-69e6-4452-874a-d2b2041cc643" (UID: "a5acb7e3-69e6-4452-874a-d2b2041cc643"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:04.730898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.730807    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:38:04.730898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.730867    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5acb7e3-69e6-4452-874a-d2b2041cc643-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5acb7e3-69e6-4452-874a-d2b2041cc643" (UID: "a5acb7e3-69e6-4452-874a-d2b2041cc643"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:38:04.731657 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.731225    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-home" (OuterVolumeSpecName: "home") pod "a5acb7e3-69e6-4452-874a-d2b2041cc643" (UID: "a5acb7e3-69e6-4452-874a-d2b2041cc643"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:04.733137 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.733109    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5acb7e3-69e6-4452-874a-d2b2041cc643-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5acb7e3-69e6-4452-874a-d2b2041cc643" (UID: "a5acb7e3-69e6-4452-874a-d2b2041cc643"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:38:04.733404 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.733381    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-dshm" (OuterVolumeSpecName: "dshm") pod "a5acb7e3-69e6-4452-874a-d2b2041cc643" (UID: "a5acb7e3-69e6-4452-874a-d2b2041cc643"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:04.733474 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.733409    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5acb7e3-69e6-4452-874a-d2b2041cc643-kube-api-access-qkcf5" (OuterVolumeSpecName: "kube-api-access-qkcf5") pod "a5acb7e3-69e6-4452-874a-d2b2041cc643" (UID: "a5acb7e3-69e6-4452-874a-d2b2041cc643"). InnerVolumeSpecName "kube-api-access-qkcf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:38:04.752475 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.752389    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5acb7e3-69e6-4452-874a-d2b2041cc643" (UID: "a5acb7e3-69e6-4452-874a-d2b2041cc643"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:04.754524 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.754487    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5acb7e3-69e6-4452-874a-d2b2041cc643" (UID: "a5acb7e3-69e6-4452-874a-d2b2041cc643"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:04.831951 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.831904    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:38:04.831951 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.831944    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:38:04.831951 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.831956    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5acb7e3-69e6-4452-874a-d2b2041cc643-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:38:04.832200 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.831969    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:38:04.832200 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.831981    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5acb7e3-69e6-4452-874a-d2b2041cc643-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:38:04.832200 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.831991    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5acb7e3-69e6-4452-874a-d2b2041cc643-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:38:04.832200 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:04.832002    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkcf5\" (UniqueName: \"kubernetes.io/projected/a5acb7e3-69e6-4452-874a-d2b2041cc643-kube-api-access-qkcf5\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:38:05.375304 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:05.375261    2574 generic.go:356] "Generic (PLEG): container finished" podID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerID="2f0d0d3fb02447f9364f55ead14f8a5aa8d915d15afaad9eda19e5a70f78961b" exitCode=0
Sep 17 18:38:05.375485 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:05.375320    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"
Sep 17 18:38:05.375485 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:05.375346    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" event={"ID":"a5acb7e3-69e6-4452-874a-d2b2041cc643","Type":"ContainerDied","Data":"2f0d0d3fb02447f9364f55ead14f8a5aa8d915d15afaad9eda19e5a70f78961b"}
Sep 17 18:38:05.375485 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:05.375404    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh" event={"ID":"a5acb7e3-69e6-4452-874a-d2b2041cc643","Type":"ContainerDied","Data":"790a30deed972c77297af31088abc5473d90e23313ada3f3476b2120c7b29715"}
Sep 17 18:38:05.375485 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:05.375426    2574 scope.go:117] "RemoveContainer" containerID="2f0d0d3fb02447f9364f55ead14f8a5aa8d915d15afaad9eda19e5a70f78961b"
Sep 17 18:38:05.386371 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:05.386350    2574 scope.go:117] "RemoveContainer" containerID="d941d1f3eb42b05646fb2f3def887d475cc1fd0501a676b134a210b27d589fc1"
Sep 17 18:38:05.394984 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:05.394960    2574 scope.go:117] "RemoveContainer" containerID="2f0d0d3fb02447f9364f55ead14f8a5aa8d915d15afaad9eda19e5a70f78961b"
Sep 17 18:38:05.395242 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:38:05.395218    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0d0d3fb02447f9364f55ead14f8a5aa8d915d15afaad9eda19e5a70f78961b\": container with ID starting with 2f0d0d3fb02447f9364f55ead14f8a5aa8d915d15afaad9eda19e5a70f78961b not found: ID does not exist" containerID="2f0d0d3fb02447f9364f55ead14f8a5aa8d915d15afaad9eda19e5a70f78961b"
Sep 17 18:38:05.395295 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:05.395252    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0d0d3fb02447f9364f55ead14f8a5aa8d915d15afaad9eda19e5a70f78961b"} err="failed to get container status \"2f0d0d3fb02447f9364f55ead14f8a5aa8d915d15afaad9eda19e5a70f78961b\": rpc error: code = NotFound desc = could not find container \"2f0d0d3fb02447f9364f55ead14f8a5aa8d915d15afaad9eda19e5a70f78961b\": container with ID starting with 2f0d0d3fb02447f9364f55ead14f8a5aa8d915d15afaad9eda19e5a70f78961b not found: ID does not exist"
Sep 17 18:38:05.395295 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:05.395271    2574 scope.go:117] "RemoveContainer" containerID="d941d1f3eb42b05646fb2f3def887d475cc1fd0501a676b134a210b27d589fc1"
Sep 17 18:38:05.395607 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:38:05.395548    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d941d1f3eb42b05646fb2f3def887d475cc1fd0501a676b134a210b27d589fc1\": container with ID starting with d941d1f3eb42b05646fb2f3def887d475cc1fd0501a676b134a210b27d589fc1 not found: ID does not exist" containerID="d941d1f3eb42b05646fb2f3def887d475cc1fd0501a676b134a210b27d589fc1"
Sep 17 18:38:05.395607 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:05.395578    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d941d1f3eb42b05646fb2f3def887d475cc1fd0501a676b134a210b27d589fc1"} err="failed to get container status \"d941d1f3eb42b05646fb2f3def887d475cc1fd0501a676b134a210b27d589fc1\": rpc error: code = NotFound desc = could not find container \"d941d1f3eb42b05646fb2f3def887d475cc1fd0501a676b134a210b27d589fc1\": container with ID starting with d941d1f3eb42b05646fb2f3def887d475cc1fd0501a676b134a210b27d589fc1 not found: ID does not exist"
Sep 17 18:38:05.399215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:05.399189    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"]
Sep 17 18:38:05.402190 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:05.402167    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76dbdf99f6-hdnsh"]
Sep 17 18:38:05.938797 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:05.938762    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" path="/var/lib/kubelet/pods/a5acb7e3-69e6-4452-874a-d2b2041cc643/volumes"
Sep 17 18:38:13.203475 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:13.203425    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:13.270716 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:13.270675    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:23.381049 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:23.381008    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:23.468573 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:23.468541    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:27.211575 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:27.211534    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"]
Sep 17 18:38:27.213526 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:27.213482    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:27.313936 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:27.313896    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:28.036831 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:28.036789    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92" podUID="8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:38:38.036573 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:38.036533    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92" podUID="8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:38:38.939745 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:38.939706    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:39.146734 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:39.146701    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:39.260558 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:39.260506    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:42.233188 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:42.233142    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92" podUID="8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e" containerName="main" containerID="cri-o://88a69d2c76de9868690f326e038feb9f921bf6e0f4f80e6465039aa9d6c986d1" gracePeriod=45
Sep 17 18:38:42.628882 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:42.628842    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:45.164881 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:45.164839    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:45.595978 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:45.595943    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:45.717152 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:45.717118    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:46.192732 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:46.192694    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:48.036973 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:48.036927    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92" podUID="8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:38:48.037449 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:48.037044    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"
Sep 17 18:38:51.900137 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:51.900094    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:52.000954 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:52.000917    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:58.036659 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:38:58.036618    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92" podUID="8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:39:08.036853 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:08.036807    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92" podUID="8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:39:12.490577 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.490485    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"
Sep 17 18:39:12.586601 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.586563    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qngdz\" (UniqueName: \"kubernetes.io/projected/8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e-kube-api-access-qngdz\") pod \"8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e\" (UID: \"8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e\") "
Sep 17 18:39:12.586821 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.586673    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e-tls-certs\") pod \"8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e\" (UID: \"8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e\") "
Sep 17 18:39:12.589085 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.589054    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e-kube-api-access-qngdz" (OuterVolumeSpecName: "kube-api-access-qngdz") pod "8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e" (UID: "8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e"). InnerVolumeSpecName "kube-api-access-qngdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:39:12.589085 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.589062    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e" (UID: "8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:39:12.687540 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.687477    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:39:12.687540 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.687537    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qngdz\" (UniqueName: \"kubernetes.io/projected/8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e-kube-api-access-qngdz\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:39:12.745152 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.745054    2574 generic.go:356] "Generic (PLEG): container finished" podID="8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e" containerID="88a69d2c76de9868690f326e038feb9f921bf6e0f4f80e6465039aa9d6c986d1" exitCode=0
Sep 17 18:39:12.745152 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.745098    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"
Sep 17 18:39:12.745152 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.745121    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92" event={"ID":"8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e","Type":"ContainerDied","Data":"88a69d2c76de9868690f326e038feb9f921bf6e0f4f80e6465039aa9d6c986d1"}
Sep 17 18:39:12.745438 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.745199    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92" event={"ID":"8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e","Type":"ContainerDied","Data":"5cd97795ee8b1c3d1f38e14573167869e3eb2955a6d686727d895b7aef198da8"}
Sep 17 18:39:12.745438 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.745229    2574 scope.go:117] "RemoveContainer" containerID="88a69d2c76de9868690f326e038feb9f921bf6e0f4f80e6465039aa9d6c986d1"
Sep 17 18:39:12.755973 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.755953    2574 scope.go:117] "RemoveContainer" containerID="88a69d2c76de9868690f326e038feb9f921bf6e0f4f80e6465039aa9d6c986d1"
Sep 17 18:39:12.756256 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:39:12.756236    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a69d2c76de9868690f326e038feb9f921bf6e0f4f80e6465039aa9d6c986d1\": container with ID starting with 88a69d2c76de9868690f326e038feb9f921bf6e0f4f80e6465039aa9d6c986d1 not found: ID does not exist" containerID="88a69d2c76de9868690f326e038feb9f921bf6e0f4f80e6465039aa9d6c986d1"
Sep 17 18:39:12.756297 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.756265    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a69d2c76de9868690f326e038feb9f921bf6e0f4f80e6465039aa9d6c986d1"} err="failed to get container status \"88a69d2c76de9868690f326e038feb9f921bf6e0f4f80e6465039aa9d6c986d1\": rpc error: code = NotFound desc = could not find container \"88a69d2c76de9868690f326e038feb9f921bf6e0f4f80e6465039aa9d6c986d1\": container with ID starting with 88a69d2c76de9868690f326e038feb9f921bf6e0f4f80e6465039aa9d6c986d1 not found: ID does not exist"
Sep 17 18:39:12.768097 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.768064    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"]
Sep 17 18:39:12.771461 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:12.771434    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepwh92"]
Sep 17 18:39:13.938979 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:13.938944    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e" path="/var/lib/kubelet/pods/8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e/volumes"
Sep 17 18:39:14.188536 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:14.188482    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:16.861602 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:16.861561    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:16.960270 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:16.960238    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:23.259531 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:23.259466    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:27.565233 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.565192    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"]
Sep 17 18:39:27.565911 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.565812    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main"
Sep 17 18:39:27.565911 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.565836    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main"
Sep 17 18:39:27.565911 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.565858    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="storage-initializer"
Sep 17 18:39:27.565911 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.565868    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="storage-initializer"
Sep 17 18:39:27.565911 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.565878    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e" containerName="main"
Sep 17 18:39:27.565911 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.565887    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e" containerName="main"
Sep 17 18:39:27.566228 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.565986    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c6d3e5f-9908-496a-9cd2-a6d21ab64c2e" containerName="main"
Sep 17 18:39:27.566228 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.565999    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5acb7e3-69e6-4452-874a-d2b2041cc643" containerName="main"
Sep 17 18:39:27.570477 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.570453    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.573098 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.573068    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:39:27.573254 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.573135    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 18:39:27.573254 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.573158    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 18:39:27.573254 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.573178    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-p7k6s\""
Sep 17 18:39:27.573254 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.573141    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 18:39:27.573886 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.573867    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 18:39:27.579690 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.579668    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"]
Sep 17 18:39:27.597783 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.597742    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:27.718655 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.718610    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-kserve-provision-location\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.718655 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.718658    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/030c0ece-cce9-42d2-b469-039fd7615451-tls-certs\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.718950 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.718714    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-tmp-dir\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.718950 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.718730    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/030c0ece-cce9-42d2-b469-039fd7615451-cabundle-cert\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.718950 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.718822    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-model-cache\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.718950 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.718876    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-dshm\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.718950 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.718923    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-home\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.719152 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.718957    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfl6w\" (UniqueName: \"kubernetes.io/projected/030c0ece-cce9-42d2-b469-039fd7615451-kube-api-access-lfl6w\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.809796 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.809743    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"]
Sep 17 18:39:27.814088 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.814060    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"
Sep 17 18:39:27.816558 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.816458    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-7wwrc\""
Sep 17 18:39:27.819743 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.819720    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-model-cache\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.819894 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.819762    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-dshm\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.819894 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.819804    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-home\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.819894 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.819835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfl6w\" (UniqueName: \"kubernetes.io/projected/030c0ece-cce9-42d2-b469-039fd7615451-kube-api-access-lfl6w\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.819894 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.819875    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-kserve-provision-location\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.820139 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.819906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/030c0ece-cce9-42d2-b469-039fd7615451-tls-certs\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.820139 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.819964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-tmp-dir\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.820139 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.819988    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/030c0ece-cce9-42d2-b469-039fd7615451-cabundle-cert\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.820310 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.820142    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-model-cache\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.820843 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.820820    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/030c0ece-cce9-42d2-b469-039fd7615451-cabundle-cert\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.821384 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.821361    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-kserve-provision-location\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.821944 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.821897    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-home\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.822170 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.822155    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-tmp-dir\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.823670 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.823629    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-dshm\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.823948 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.823923    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/030c0ece-cce9-42d2-b469-039fd7615451-tls-certs\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.824884 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.824859    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"]
Sep 17 18:39:27.833587 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.833536    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfl6w\" (UniqueName: \"kubernetes.io/projected/030c0ece-cce9-42d2-b469-039fd7615451-kube-api-access-lfl6w\") pod \"stop-feature-test-kserve-66fd6b68fd-w4fwv\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.849000 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.848967    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:27.882164 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.882130    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:27.920603 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.920558    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csh4x\" (UniqueName: \"kubernetes.io/projected/bf57f11b-35a9-446a-bedf-6e4901b4aa24-kube-api-access-csh4x\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks\" (UID: \"bf57f11b-35a9-446a-bedf-6e4901b4aa24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"
Sep 17 18:39:27.920813 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:27.920615    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf57f11b-35a9-446a-bedf-6e4901b4aa24-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks\" (UID: \"bf57f11b-35a9-446a-bedf-6e4901b4aa24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"
Sep 17 18:39:28.021842 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:28.021795    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csh4x\" (UniqueName: \"kubernetes.io/projected/bf57f11b-35a9-446a-bedf-6e4901b4aa24-kube-api-access-csh4x\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks\" (UID: \"bf57f11b-35a9-446a-bedf-6e4901b4aa24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"
Sep 17 18:39:28.022058 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:28.021868    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf57f11b-35a9-446a-bedf-6e4901b4aa24-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks\" (UID: \"bf57f11b-35a9-446a-bedf-6e4901b4aa24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"
Sep 17 18:39:28.024693 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:28.024666    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf57f11b-35a9-446a-bedf-6e4901b4aa24-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks\" (UID: \"bf57f11b-35a9-446a-bedf-6e4901b4aa24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"
Sep 17 18:39:28.033579 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:28.033555    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csh4x\" (UniqueName: \"kubernetes.io/projected/bf57f11b-35a9-446a-bedf-6e4901b4aa24-kube-api-access-csh4x\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks\" (UID: \"bf57f11b-35a9-446a-bedf-6e4901b4aa24\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"
Sep 17 18:39:28.148199 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:28.148114    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"
Sep 17 18:39:28.160990 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:28.160963    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"]
Sep 17 18:39:28.163132 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:39:28.163107    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030c0ece_cce9_42d2_b469_039fd7615451.slice/crio-c9c9fe4b177b1db7dd100e3c60d6f1166c0add6ed986fa4b23a9c4b280dd6ff5 WatchSource:0}: Error finding container c9c9fe4b177b1db7dd100e3c60d6f1166c0add6ed986fa4b23a9c4b280dd6ff5: Status 404 returned error can't find the container with id c9c9fe4b177b1db7dd100e3c60d6f1166c0add6ed986fa4b23a9c4b280dd6ff5
Sep 17 18:39:28.429871 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:28.429836    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"]
Sep 17 18:39:28.429989 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:39:28.429837    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf57f11b_35a9_446a_bedf_6e4901b4aa24.slice/crio-3f78d943b932eb607cd36350036b943a493ed0b6f07d3412b0387579bd5a7fd0 WatchSource:0}: Error finding container 3f78d943b932eb607cd36350036b943a493ed0b6f07d3412b0387579bd5a7fd0: Status 404 returned error can't find the container with id 3f78d943b932eb607cd36350036b943a493ed0b6f07d3412b0387579bd5a7fd0
Sep 17 18:39:28.841864 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:28.841819    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" event={"ID":"030c0ece-cce9-42d2-b469-039fd7615451","Type":"ContainerStarted","Data":"752f04920198a3a0f1c55c05daa457650bb069729a0c65e88ffb19063bb0c3eb"}
Sep 17 18:39:28.842309 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:28.841889    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" event={"ID":"030c0ece-cce9-42d2-b469-039fd7615451","Type":"ContainerStarted","Data":"c9c9fe4b177b1db7dd100e3c60d6f1166c0add6ed986fa4b23a9c4b280dd6ff5"}
Sep 17 18:39:28.843599 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:28.843573    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks" event={"ID":"bf57f11b-35a9-446a-bedf-6e4901b4aa24","Type":"ContainerStarted","Data":"7aa3ebcb5395c37bd938ea46907a4ff2c40c29d718a1b52c933a27fd6669a3d0"}
Sep 17 18:39:28.843689 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:28.843619    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks" event={"ID":"bf57f11b-35a9-446a-bedf-6e4901b4aa24","Type":"ContainerStarted","Data":"3f78d943b932eb607cd36350036b943a493ed0b6f07d3412b0387579bd5a7fd0"}
Sep 17 18:39:28.843689 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:28.843657    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"
Sep 17 18:39:28.885863 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:28.885811    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks" podStartSLOduration=1.885795608 podStartE2EDuration="1.885795608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:28.884883693 +0000 UTC m=+2539.185706019" watchObservedRunningTime="2026-09-17 18:39:28.885795608 +0000 UTC m=+2539.186617933"
Sep 17 18:39:29.904794 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:29.904743    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:30.004907 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:30.004818    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:30.855508 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:30.855404    2574 generic.go:356] "Generic (PLEG): container finished" podID="030c0ece-cce9-42d2-b469-039fd7615451" containerID="752f04920198a3a0f1c55c05daa457650bb069729a0c65e88ffb19063bb0c3eb" exitCode=0
Sep 17 18:39:30.855508 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:30.855481    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" event={"ID":"030c0ece-cce9-42d2-b469-039fd7615451","Type":"ContainerDied","Data":"752f04920198a3a0f1c55c05daa457650bb069729a0c65e88ffb19063bb0c3eb"}
Sep 17 18:39:31.863433 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:31.863400    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" event={"ID":"030c0ece-cce9-42d2-b469-039fd7615451","Type":"ContainerStarted","Data":"0accb51f2f2710e59ed77f978ae59e880ed36ec64fde50676883c9efd46f4ece"}
Sep 17 18:39:31.882549 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:31.882497    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:31.887648 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:31.887586    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" podStartSLOduration=2.544145188 podStartE2EDuration="4.887563771s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.343418583s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:31.886837665 +0000 UTC m=+2542.187659989" watchObservedRunningTime="2026-09-17 18:39:31.887563771 +0000 UTC m=+2542.188386104"
Sep 17 18:39:36.172772 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:36.172734    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:37.883118 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:37.883079    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:39:37.885045 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:37.885015    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 18:39:40.095085 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:40.095040    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:47.883020 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:47.882974    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 18:39:57.883132 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:57.883083    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 18:39:59.850632 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:39:59.850599    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"
Sep 17 18:40:07.882818 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:40:07.882765    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 18:40:17.883311 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:40:17.883263    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 18:40:27.883102 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:40:27.882999    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 18:40:37.882961 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:40:37.882916    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 18:40:47.883274 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:40:47.883219    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 18:40:49.908117 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:40:49.908074    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:51.280996 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:40:51.280951    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:57.882892 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:40:57.882829    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 18:41:07.882555 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:07.882497    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 18:41:17.882914 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:17.882861    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 18:41:27.893430 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:27.893396    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:41:27.901892 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:27.901862    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:41:29.683064 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:29.683020    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"]
Sep 17 18:41:29.709705 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:29.709671    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:29.929889 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:29.929849    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"]
Sep 17 18:41:30.048561 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:30.048507    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:30.157542 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:30.157493    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:39.850632 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:39.850585    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks" podUID="bf57f11b-35a9-446a-bedf-6e4901b4aa24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:39.931969 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:39.931929    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:40.028297 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:40.028261    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:44.707687 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:44.707648    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main" containerID="cri-o://0accb51f2f2710e59ed77f978ae59e880ed36ec64fde50676883c9efd46f4ece" gracePeriod=45
Sep 17 18:41:44.894122 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:44.894074    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 18:41:44.950494 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:44.950430    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks" podUID="bf57f11b-35a9-446a-bedf-6e4901b4aa24" containerName="main" containerID="cri-o://7aa3ebcb5395c37bd938ea46907a4ff2c40c29d718a1b52c933a27fd6669a3d0" gracePeriod=45
Sep 17 18:41:44.965976 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:44.965886    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:45.384710 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.384679    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:45.469982 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.469953    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:41:45.544326 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.544294    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-tmp-dir\") pod \"030c0ece-cce9-42d2-b469-039fd7615451\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") "
Sep 17 18:41:45.544534 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.544348    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-kserve-provision-location\") pod \"030c0ece-cce9-42d2-b469-039fd7615451\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") "
Sep 17 18:41:45.544534 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.544378    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-home\") pod \"030c0ece-cce9-42d2-b469-039fd7615451\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") "
Sep 17 18:41:45.544534 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.544403    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-model-cache\") pod \"030c0ece-cce9-42d2-b469-039fd7615451\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") "
Sep 17 18:41:45.544534 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.544433    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/030c0ece-cce9-42d2-b469-039fd7615451-tls-certs\") pod \"030c0ece-cce9-42d2-b469-039fd7615451\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") "
Sep 17 18:41:45.544534 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.544504    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-dshm\") pod \"030c0ece-cce9-42d2-b469-039fd7615451\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") "
Sep 17 18:41:45.544810 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.544563    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/030c0ece-cce9-42d2-b469-039fd7615451-cabundle-cert\") pod \"030c0ece-cce9-42d2-b469-039fd7615451\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") "
Sep 17 18:41:45.544810 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.544613    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfl6w\" (UniqueName: \"kubernetes.io/projected/030c0ece-cce9-42d2-b469-039fd7615451-kube-api-access-lfl6w\") pod \"030c0ece-cce9-42d2-b469-039fd7615451\" (UID: \"030c0ece-cce9-42d2-b469-039fd7615451\") "
Sep 17 18:41:45.545008 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.544806    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-model-cache" (OuterVolumeSpecName: "model-cache") pod "030c0ece-cce9-42d2-b469-039fd7615451" (UID: "030c0ece-cce9-42d2-b469-039fd7615451"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:45.545148 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.545074    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030c0ece-cce9-42d2-b469-039fd7615451-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "030c0ece-cce9-42d2-b469-039fd7615451" (UID: "030c0ece-cce9-42d2-b469-039fd7615451"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:45.545413 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.545387    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-home" (OuterVolumeSpecName: "home") pod "030c0ece-cce9-42d2-b469-039fd7615451" (UID: "030c0ece-cce9-42d2-b469-039fd7615451"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:45.547396 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.547373    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030c0ece-cce9-42d2-b469-039fd7615451-kube-api-access-lfl6w" (OuterVolumeSpecName: "kube-api-access-lfl6w") pod "030c0ece-cce9-42d2-b469-039fd7615451" (UID: "030c0ece-cce9-42d2-b469-039fd7615451"). InnerVolumeSpecName "kube-api-access-lfl6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:45.547565 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.547543    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-dshm" (OuterVolumeSpecName: "dshm") pod "030c0ece-cce9-42d2-b469-039fd7615451" (UID: "030c0ece-cce9-42d2-b469-039fd7615451"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:45.547645 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.547554    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/030c0ece-cce9-42d2-b469-039fd7615451-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "030c0ece-cce9-42d2-b469-039fd7615451" (UID: "030c0ece-cce9-42d2-b469-039fd7615451"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:45.564260 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.564226    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "030c0ece-cce9-42d2-b469-039fd7615451" (UID: "030c0ece-cce9-42d2-b469-039fd7615451"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:45.564356 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.564259    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "030c0ece-cce9-42d2-b469-039fd7615451" (UID: "030c0ece-cce9-42d2-b469-039fd7615451"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:45.645826 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.645776    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:41:45.645826 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.645817    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:41:45.645826 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.645828    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/030c0ece-cce9-42d2-b469-039fd7615451-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:41:45.645826 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.645836    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:41:45.646098 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.645844    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/030c0ece-cce9-42d2-b469-039fd7615451-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:41:45.646098 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.645852    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfl6w\" (UniqueName: \"kubernetes.io/projected/030c0ece-cce9-42d2-b469-039fd7615451-kube-api-access-lfl6w\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:41:45.646098 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.645881    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:41:45.646098 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.645890    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/030c0ece-cce9-42d2-b469-039fd7615451-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:41:45.655604 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.655570    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:45.663577 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.663553    2574 generic.go:356] "Generic (PLEG): container finished" podID="030c0ece-cce9-42d2-b469-039fd7615451" containerID="0accb51f2f2710e59ed77f978ae59e880ed36ec64fde50676883c9efd46f4ece" exitCode=0
Sep 17 18:41:45.663708 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.663608    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"
Sep 17 18:41:45.663708 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.663633    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" event={"ID":"030c0ece-cce9-42d2-b469-039fd7615451","Type":"ContainerDied","Data":"0accb51f2f2710e59ed77f978ae59e880ed36ec64fde50676883c9efd46f4ece"}
Sep 17 18:41:45.663708 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.663696    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv" event={"ID":"030c0ece-cce9-42d2-b469-039fd7615451","Type":"ContainerDied","Data":"c9c9fe4b177b1db7dd100e3c60d6f1166c0add6ed986fa4b23a9c4b280dd6ff5"}
Sep 17 18:41:45.663842 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.663725    2574 scope.go:117] "RemoveContainer" containerID="0accb51f2f2710e59ed77f978ae59e880ed36ec64fde50676883c9efd46f4ece"
Sep 17 18:41:45.673997 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.673981    2574 scope.go:117] "RemoveContainer" containerID="752f04920198a3a0f1c55c05daa457650bb069729a0c65e88ffb19063bb0c3eb"
Sep 17 18:41:45.682580 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.682559    2574 scope.go:117] "RemoveContainer" containerID="0accb51f2f2710e59ed77f978ae59e880ed36ec64fde50676883c9efd46f4ece"
Sep 17 18:41:45.682851 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:41:45.682833    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0accb51f2f2710e59ed77f978ae59e880ed36ec64fde50676883c9efd46f4ece\": container with ID starting with 0accb51f2f2710e59ed77f978ae59e880ed36ec64fde50676883c9efd46f4ece not found: ID does not exist" containerID="0accb51f2f2710e59ed77f978ae59e880ed36ec64fde50676883c9efd46f4ece"
Sep 17 18:41:45.682901 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.682860    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0accb51f2f2710e59ed77f978ae59e880ed36ec64fde50676883c9efd46f4ece"} err="failed to get container status \"0accb51f2f2710e59ed77f978ae59e880ed36ec64fde50676883c9efd46f4ece\": rpc error: code = NotFound desc = could not find container \"0accb51f2f2710e59ed77f978ae59e880ed36ec64fde50676883c9efd46f4ece\": container with ID starting with 0accb51f2f2710e59ed77f978ae59e880ed36ec64fde50676883c9efd46f4ece not found: ID does not exist"
Sep 17 18:41:45.682901 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.682877    2574 scope.go:117] "RemoveContainer" containerID="752f04920198a3a0f1c55c05daa457650bb069729a0c65e88ffb19063bb0c3eb"
Sep 17 18:41:45.683108 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:41:45.683093    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"752f04920198a3a0f1c55c05daa457650bb069729a0c65e88ffb19063bb0c3eb\": container with ID starting with 752f04920198a3a0f1c55c05daa457650bb069729a0c65e88ffb19063bb0c3eb not found: ID does not exist" containerID="752f04920198a3a0f1c55c05daa457650bb069729a0c65e88ffb19063bb0c3eb"
Sep 17 18:41:45.683144 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.683114    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"752f04920198a3a0f1c55c05daa457650bb069729a0c65e88ffb19063bb0c3eb"} err="failed to get container status \"752f04920198a3a0f1c55c05daa457650bb069729a0c65e88ffb19063bb0c3eb\": rpc error: code = NotFound desc = could not find container \"752f04920198a3a0f1c55c05daa457650bb069729a0c65e88ffb19063bb0c3eb\": container with ID starting with 752f04920198a3a0f1c55c05daa457650bb069729a0c65e88ffb19063bb0c3eb not found: ID does not exist"
Sep 17 18:41:45.687078 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.687056    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"]
Sep 17 18:41:45.690402 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.690376    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-w4fwv"]
Sep 17 18:41:45.939333 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:45.939251    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="030c0ece-cce9-42d2-b469-039fd7615451" path="/var/lib/kubelet/pods/030c0ece-cce9-42d2-b469-039fd7615451/volumes"
Sep 17 18:41:49.851277 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:49.851231    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks" podUID="bf57f11b-35a9-446a-bedf-6e4901b4aa24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:56.495673 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:56.495639    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:59.850850 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:59.850812    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks" podUID="bf57f11b-35a9-446a-bedf-6e4901b4aa24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:59.851371 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:41:59.850943    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"
Sep 17 18:42:06.017796 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:06.017760    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:07.833789 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:07.833751    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:09.851113 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:09.851062    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks" podUID="bf57f11b-35a9-446a-bedf-6e4901b4aa24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:11.820801 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.820754    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"]
Sep 17 18:42:11.821386 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.821367    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="storage-initializer"
Sep 17 18:42:11.821454 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.821390    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="storage-initializer"
Sep 17 18:42:11.821454 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.821418    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main"
Sep 17 18:42:11.821454 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.821427    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main"
Sep 17 18:42:11.821586 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.821526    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="030c0ece-cce9-42d2-b469-039fd7615451" containerName="main"
Sep 17 18:42:11.825098 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.825070    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.827710 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.827683    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 18:42:11.827830 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.827691    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-zhdlm\""
Sep 17 18:42:11.827830 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.827793    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 18:42:11.828087 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.828063    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:42:11.828188 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.828154    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-gx8l6\""
Sep 17 18:42:11.828188 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.828153    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 18:42:11.828714 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.828697    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 18:42:11.832136 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.832100    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"]
Sep 17 18:42:11.836116 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.836095    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.840094 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.840072    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"]
Sep 17 18:42:11.855179 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.855149    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"]
Sep 17 18:42:11.870225 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870157    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.870225 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870200    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94be6141-84e1-4e00-9334-910d8bbad626-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.870422 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870233    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94be6141-84e1-4e00-9334-910d8bbad626-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.870422 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870265    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.870422 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870286    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.870422 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870327    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.870422 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870378    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.870666 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870436    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.870666 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870474    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.870666 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870492    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tzrk\" (UniqueName: \"kubernetes.io/projected/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-kube-api-access-8tzrk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.870666 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870552    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.870666 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870582    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.870666 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870632    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.871133 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870678    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf6bq\" (UniqueName: \"kubernetes.io/projected/94be6141-84e1-4e00-9334-910d8bbad626-kube-api-access-hf6bq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.871133 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.871133 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.870735    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.888200 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.888173    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:11.971266 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971214    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tzrk\" (UniqueName: \"kubernetes.io/projected/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-kube-api-access-8tzrk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.971266 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971266    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.971580 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971298    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.971580 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971321    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.971580 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971346    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf6bq\" (UniqueName: \"kubernetes.io/projected/94be6141-84e1-4e00-9334-910d8bbad626-kube-api-access-hf6bq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.971580 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971369    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.971580 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971435    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.971863 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971621    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.971863 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971666    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94be6141-84e1-4e00-9334-910d8bbad626-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.971863 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971699    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94be6141-84e1-4e00-9334-910d8bbad626-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.971863 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.971863 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971751    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.971863 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971775    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.971863 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971811    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.971863 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971826    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.972243 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971863    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.972243 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971873    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.972243 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971951    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.972243 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.971987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.972460 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.972374    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.972584 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.972562    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.972662 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.972593    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94be6141-84e1-4e00-9334-910d8bbad626-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.972721 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.972685    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.972721 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.972704    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.972823 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.972752    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.972870 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.972847    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.974031 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.974003    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.974557 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.974535    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.975023 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.975001    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94be6141-84e1-4e00-9334-910d8bbad626-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.975238 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.975223    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:11.995013 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.994978    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf6bq\" (UniqueName: \"kubernetes.io/projected/94be6141-84e1-4e00-9334-910d8bbad626-kube-api-access-hf6bq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:11.995116 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:11.994980    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tzrk\" (UniqueName: \"kubernetes.io/projected/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-kube-api-access-8tzrk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:12.056488 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:12.056459    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:42:12.070191 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:12.070159    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:42:12.138464 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:12.138375    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:12.148172 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:12.148144    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:12.212580 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:12.212436    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:12.428902 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:12.428876    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"]
Sep 17 18:42:12.430553 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:42:12.430501    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94be6141_84e1_4e00_9334_910d8bbad626.slice/crio-d08c889881f70a97082cd03d1b9dcf74263599007ddb99a1d4c2af335eac87a5 WatchSource:0}: Error finding container d08c889881f70a97082cd03d1b9dcf74263599007ddb99a1d4c2af335eac87a5: Status 404 returned error can't find the container with id d08c889881f70a97082cd03d1b9dcf74263599007ddb99a1d4c2af335eac87a5
Sep 17 18:42:12.433075 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:12.433055    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:42:12.435182 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:12.435148    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"]
Sep 17 18:42:12.438537 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:42:12.437493    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb3dec5_1d18_4e94_bb3e_5fd1f2ae9b15.slice/crio-980657281d307e11ae685a53ba4c4a32beb20377d09b3d83f071dfd610efc5d3 WatchSource:0}: Error finding container 980657281d307e11ae685a53ba4c4a32beb20377d09b3d83f071dfd610efc5d3: Status 404 returned error can't find the container with id 980657281d307e11ae685a53ba4c4a32beb20377d09b3d83f071dfd610efc5d3
Sep 17 18:42:12.822203 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:12.822162    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" event={"ID":"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15","Type":"ContainerStarted","Data":"28018769096e2f3adcfb7228e607782f717e77cf4cdcfec092a8b973a5610fce"}
Sep 17 18:42:12.822716 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:12.822235    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" event={"ID":"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15","Type":"ContainerStarted","Data":"980657281d307e11ae685a53ba4c4a32beb20377d09b3d83f071dfd610efc5d3"}
Sep 17 18:42:12.823556 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:12.823532    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" event={"ID":"94be6141-84e1-4e00-9334-910d8bbad626","Type":"ContainerStarted","Data":"d08c889881f70a97082cd03d1b9dcf74263599007ddb99a1d4c2af335eac87a5"}
Sep 17 18:42:14.295029 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:14.295001    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:14.835904 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:14.835862    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" event={"ID":"94be6141-84e1-4e00-9334-910d8bbad626","Type":"ContainerStarted","Data":"5151fb007a3d5454c013a7510b3a4f769fa8fc897464610742552943ddd23ecb"}
Sep 17 18:42:14.836091 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:14.836008    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:15.204946 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.204912    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"
Sep 17 18:42:15.305164 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.305131    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csh4x\" (UniqueName: \"kubernetes.io/projected/bf57f11b-35a9-446a-bedf-6e4901b4aa24-kube-api-access-csh4x\") pod \"bf57f11b-35a9-446a-bedf-6e4901b4aa24\" (UID: \"bf57f11b-35a9-446a-bedf-6e4901b4aa24\") "
Sep 17 18:42:15.305656 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.305218    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf57f11b-35a9-446a-bedf-6e4901b4aa24-tls-certs\") pod \"bf57f11b-35a9-446a-bedf-6e4901b4aa24\" (UID: \"bf57f11b-35a9-446a-bedf-6e4901b4aa24\") "
Sep 17 18:42:15.307612 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.307580    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf57f11b-35a9-446a-bedf-6e4901b4aa24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf57f11b-35a9-446a-bedf-6e4901b4aa24" (UID: "bf57f11b-35a9-446a-bedf-6e4901b4aa24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:15.307723 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.307629    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf57f11b-35a9-446a-bedf-6e4901b4aa24-kube-api-access-csh4x" (OuterVolumeSpecName: "kube-api-access-csh4x") pod "bf57f11b-35a9-446a-bedf-6e4901b4aa24" (UID: "bf57f11b-35a9-446a-bedf-6e4901b4aa24"). InnerVolumeSpecName "kube-api-access-csh4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:15.406042 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.405955    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csh4x\" (UniqueName: \"kubernetes.io/projected/bf57f11b-35a9-446a-bedf-6e4901b4aa24-kube-api-access-csh4x\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:42:15.406042 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.405988    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf57f11b-35a9-446a-bedf-6e4901b4aa24-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:42:15.842124 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.842090    2574 generic.go:356] "Generic (PLEG): container finished" podID="bf57f11b-35a9-446a-bedf-6e4901b4aa24" containerID="7aa3ebcb5395c37bd938ea46907a4ff2c40c29d718a1b52c933a27fd6669a3d0" exitCode=0
Sep 17 18:42:15.842297 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.842130    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"
Sep 17 18:42:15.842297 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.842170    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks" event={"ID":"bf57f11b-35a9-446a-bedf-6e4901b4aa24","Type":"ContainerDied","Data":"7aa3ebcb5395c37bd938ea46907a4ff2c40c29d718a1b52c933a27fd6669a3d0"}
Sep 17 18:42:15.842297 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.842222    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks" event={"ID":"bf57f11b-35a9-446a-bedf-6e4901b4aa24","Type":"ContainerDied","Data":"3f78d943b932eb607cd36350036b943a493ed0b6f07d3412b0387579bd5a7fd0"}
Sep 17 18:42:15.842297 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.842249    2574 scope.go:117] "RemoveContainer" containerID="7aa3ebcb5395c37bd938ea46907a4ff2c40c29d718a1b52c933a27fd6669a3d0"
Sep 17 18:42:15.843924 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.843904    2574 generic.go:356] "Generic (PLEG): container finished" podID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerID="28018769096e2f3adcfb7228e607782f717e77cf4cdcfec092a8b973a5610fce" exitCode=0
Sep 17 18:42:15.844004 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.843970    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" event={"ID":"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15","Type":"ContainerDied","Data":"28018769096e2f3adcfb7228e607782f717e77cf4cdcfec092a8b973a5610fce"}
Sep 17 18:42:15.846010 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.845990    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" event={"ID":"94be6141-84e1-4e00-9334-910d8bbad626","Type":"ContainerStarted","Data":"119eee43b4e9a0ff106f1d5756a06e59e7ba25246ea1e6232feccf08d335510f"}
Sep 17 18:42:15.853098 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.853068    2574 scope.go:117] "RemoveContainer" containerID="7aa3ebcb5395c37bd938ea46907a4ff2c40c29d718a1b52c933a27fd6669a3d0"
Sep 17 18:42:15.853393 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:42:15.853375    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa3ebcb5395c37bd938ea46907a4ff2c40c29d718a1b52c933a27fd6669a3d0\": container with ID starting with 7aa3ebcb5395c37bd938ea46907a4ff2c40c29d718a1b52c933a27fd6669a3d0 not found: ID does not exist" containerID="7aa3ebcb5395c37bd938ea46907a4ff2c40c29d718a1b52c933a27fd6669a3d0"
Sep 17 18:42:15.853449 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.853402    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa3ebcb5395c37bd938ea46907a4ff2c40c29d718a1b52c933a27fd6669a3d0"} err="failed to get container status \"7aa3ebcb5395c37bd938ea46907a4ff2c40c29d718a1b52c933a27fd6669a3d0\": rpc error: code = NotFound desc = could not find container \"7aa3ebcb5395c37bd938ea46907a4ff2c40c29d718a1b52c933a27fd6669a3d0\": container with ID starting with 7aa3ebcb5395c37bd938ea46907a4ff2c40c29d718a1b52c933a27fd6669a3d0 not found: ID does not exist"
Sep 17 18:42:15.900216 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.900186    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"]
Sep 17 18:42:15.905486 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.905460    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-62dks"]
Sep 17 18:42:15.939672 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:15.939636    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf57f11b-35a9-446a-bedf-6e4901b4aa24" path="/var/lib/kubelet/pods/bf57f11b-35a9-446a-bedf-6e4901b4aa24/volumes"
Sep 17 18:42:16.855399 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:16.855345    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" event={"ID":"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15","Type":"ContainerStarted","Data":"cceb8da82be046afe872567404906dc090842cff88aac94f5bb639a9d9673ce3"}
Sep 17 18:42:16.880706 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:16.880646    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podStartSLOduration=3.457619961 podStartE2EDuration="5.880629448s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.423009487s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:16.878968539 +0000 UTC m=+2707.179790863" watchObservedRunningTime="2026-09-17 18:42:16.880629448 +0000 UTC m=+2707.181451812"
Sep 17 18:42:17.452833 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:17.452771    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:17.864612 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:17.864571    2574 generic.go:356] "Generic (PLEG): container finished" podID="94be6141-84e1-4e00-9334-910d8bbad626" containerID="119eee43b4e9a0ff106f1d5756a06e59e7ba25246ea1e6232feccf08d335510f" exitCode=0
Sep 17 18:42:17.865005 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:17.864638    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" event={"ID":"94be6141-84e1-4e00-9334-910d8bbad626","Type":"ContainerDied","Data":"119eee43b4e9a0ff106f1d5756a06e59e7ba25246ea1e6232feccf08d335510f"}
Sep 17 18:42:18.875152 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:18.875107    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" event={"ID":"94be6141-84e1-4e00-9334-910d8bbad626","Type":"ContainerStarted","Data":"ae5d8cb3e85fe4391a5a012438ebfa02631cf439eeffe444ec67ff6c0a5e4fe2"}
Sep 17 18:42:18.899359 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:18.899304    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podStartSLOduration=3.678058623 podStartE2EDuration="7.899286238s" totalImagesPullingTime="1.83349835s" totalInitContainerRuntime="2.387729265s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:18.897931691 +0000 UTC m=+2709.198754042" watchObservedRunningTime="2026-09-17 18:42:18.899286238 +0000 UTC m=+2709.200108562"
Sep 17 18:42:21.134108 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:21.134051    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:22.139170 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:22.139129    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:22.139791 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:22.139219    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:22.140612 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:22.140582    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:42:22.148446 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:22.148418    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:22.148593 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:22.148478    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:42:22.149694 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:22.149656    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:42:27.146049 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:27.146013    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:28.762125 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:28.762084    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:28.826903 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:28.826859    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:32.138904 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:32.138856    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:42:32.149072 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:32.149032    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:42:32.157186 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:32.157159    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:42:42.139328 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:42.139272    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:42:42.148706 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:42.148662    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:42:52.139540 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:52.139467    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:42:52.149420 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:42:52.149382    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:43:02.139643 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:43:02.139588    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:43:02.148706 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:43:02.148669    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:43:12.139760 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:43:12.139702    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:43:12.148962 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:43:12.148919    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:43:22.139574 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:43:22.139464    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:43:22.148643 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:43:22.148604    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:43:32.139738 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:43:32.139689    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:43:32.148655 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:43:32.148612    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:43:42.139493 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:43:42.139440    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:43:42.149562 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:43:42.149507    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:43:47.385779 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:43:47.385742    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:47.488290 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:43:47.488252    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:52.139021 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:43:52.138962    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:43:52.150230 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:43:52.150183    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:44:02.139280 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:02.139234    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:44:02.150153 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:02.150113    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:44:12.139238 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:12.139183    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:44:12.149235 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:12.149187    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:44:22.138846 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:22.138789    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:44:22.150104 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:22.150060    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:44:26.406600 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:26.406563    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:26.508083 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:26.508029    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:32.139108 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:32.139052    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:44:32.148668 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:32.148620    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:44:37.108881 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:37.108835    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:37.208567 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:37.208509    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:37.314361 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:37.314312    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:42.139361 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:42.139309    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Sep 17 18:44:42.149200 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:42.149158    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 18:44:43.293899 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:43.293860    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:47.734131 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:47.734087    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:49.458134 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:49.458098    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:49.558695 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:49.558652    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:51.133108 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:51.133023    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:51.237816 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:51.237780    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:52.149409 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:52.149370    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:44:52.159958 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:52.159927    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:44:52.162623 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:52.162597    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:44:52.168377 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:44:52.168353    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:45:03.927084 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:03.927046    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"]
Sep 17 18:45:03.941332 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:03.941302    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"]
Sep 17 18:45:03.942215 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:03.942181    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:04.031929 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:04.031892    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:16.648938 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:16.648899    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:16.720502 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:16.720466    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:18.953641 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:18.953597    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main" containerID="cri-o://cceb8da82be046afe872567404906dc090842cff88aac94f5bb639a9d9673ce3" gracePeriod=45
Sep 17 18:45:18.965897 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:18.965862    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main" containerID="cri-o://ae5d8cb3e85fe4391a5a012438ebfa02631cf439eeffe444ec67ff6c0a5e4fe2" gracePeriod=45
Sep 17 18:45:19.884323 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:19.884284    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="llm-d-routing-sidecar" containerID="cri-o://5151fb007a3d5454c013a7510b3a4f769fa8fc897464610742552943ddd23ecb" gracePeriod=45
Sep 17 18:45:19.988319 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:19.988275    2574 generic.go:356] "Generic (PLEG): container finished" podID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerID="cceb8da82be046afe872567404906dc090842cff88aac94f5bb639a9d9673ce3" exitCode=0
Sep 17 18:45:19.988967 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:19.988375    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" event={"ID":"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15","Type":"ContainerDied","Data":"cceb8da82be046afe872567404906dc090842cff88aac94f5bb639a9d9673ce3"}
Sep 17 18:45:19.991622 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:19.991573    2574 generic.go:356] "Generic (PLEG): container finished" podID="94be6141-84e1-4e00-9334-910d8bbad626" containerID="ae5d8cb3e85fe4391a5a012438ebfa02631cf439eeffe444ec67ff6c0a5e4fe2" exitCode=0
Sep 17 18:45:19.991809 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:19.991617    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" event={"ID":"94be6141-84e1-4e00-9334-910d8bbad626","Type":"ContainerDied","Data":"ae5d8cb3e85fe4391a5a012438ebfa02631cf439eeffe444ec67ff6c0a5e4fe2"}
Sep 17 18:45:20.052137 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.052064    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:45:20.114948 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.114903    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-tmp-dir\") pod \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") "
Sep 17 18:45:20.115141 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.114964    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tzrk\" (UniqueName: \"kubernetes.io/projected/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-kube-api-access-8tzrk\") pod \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") "
Sep 17 18:45:20.115141 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.115033    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-tls-certs\") pod \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") "
Sep 17 18:45:20.115141 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.115066    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-home\") pod \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") "
Sep 17 18:45:20.115141 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.115105    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-kserve-provision-location\") pod \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") "
Sep 17 18:45:20.116488 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.115490    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-cabundle-cert\") pod \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") "
Sep 17 18:45:20.116488 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.115561    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-dshm\") pod \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") "
Sep 17 18:45:20.116488 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.115598    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-model-cache\") pod \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\" (UID: \"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15\") "
Sep 17 18:45:20.116488 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.115949    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-home" (OuterVolumeSpecName: "home") pod "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" (UID: "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:20.116488 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.116452    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-model-cache" (OuterVolumeSpecName: "model-cache") pod "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" (UID: "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:20.116488 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.116471    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" (UID: "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:45:20.118270 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.118235    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" (UID: "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:45:20.118814 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.118790    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" (UID: "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:20.119252 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.119219    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-kube-api-access-8tzrk" (OuterVolumeSpecName: "kube-api-access-8tzrk") pod "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" (UID: "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15"). InnerVolumeSpecName "kube-api-access-8tzrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:20.119252 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.119240    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-dshm" (OuterVolumeSpecName: "dshm") pod "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" (UID: "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:20.129680 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.129641    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" (UID: "dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:20.178234 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.178205    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:45:20.216871 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.216834    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-home\") pod \"94be6141-84e1-4e00-9334-910d8bbad626\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") "
Sep 17 18:45:20.217100 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.216917    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94be6141-84e1-4e00-9334-910d8bbad626-tls-certs\") pod \"94be6141-84e1-4e00-9334-910d8bbad626\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") "
Sep 17 18:45:20.217100 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.216945    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-tmp-dir\") pod \"94be6141-84e1-4e00-9334-910d8bbad626\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") "
Sep 17 18:45:20.217100 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.216961    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-model-cache\") pod \"94be6141-84e1-4e00-9334-910d8bbad626\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") "
Sep 17 18:45:20.217100 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.216990    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-dshm\") pod \"94be6141-84e1-4e00-9334-910d8bbad626\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") "
Sep 17 18:45:20.217100 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.217057    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94be6141-84e1-4e00-9334-910d8bbad626-cabundle-cert\") pod \"94be6141-84e1-4e00-9334-910d8bbad626\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") "
Sep 17 18:45:20.217100 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.217086    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-kserve-provision-location\") pod \"94be6141-84e1-4e00-9334-910d8bbad626\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") "
Sep 17 18:45:20.217431 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.217154    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hf6bq\" (UniqueName: \"kubernetes.io/projected/94be6141-84e1-4e00-9334-910d8bbad626-kube-api-access-hf6bq\") pod \"94be6141-84e1-4e00-9334-910d8bbad626\" (UID: \"94be6141-84e1-4e00-9334-910d8bbad626\") "
Sep 17 18:45:20.217431 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.217319    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-model-cache" (OuterVolumeSpecName: "model-cache") pod "94be6141-84e1-4e00-9334-910d8bbad626" (UID: "94be6141-84e1-4e00-9334-910d8bbad626"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:20.217572 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.217472    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.217572 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.217491    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.217572 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.217505    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.217572 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.217549    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.217572 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.217564    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.217830 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.217576    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tzrk\" (UniqueName: \"kubernetes.io/projected/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-kube-api-access-8tzrk\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.217830 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.217592    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.217830 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.217606    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.217830 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.217619    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.217830 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.217773    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94be6141-84e1-4e00-9334-910d8bbad626-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "94be6141-84e1-4e00-9334-910d8bbad626" (UID: "94be6141-84e1-4e00-9334-910d8bbad626"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:45:20.218114 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.218088    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-home" (OuterVolumeSpecName: "home") pod "94be6141-84e1-4e00-9334-910d8bbad626" (UID: "94be6141-84e1-4e00-9334-910d8bbad626"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:20.220199 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.220162    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94be6141-84e1-4e00-9334-910d8bbad626-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94be6141-84e1-4e00-9334-910d8bbad626" (UID: "94be6141-84e1-4e00-9334-910d8bbad626"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:45:20.220199 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.220168    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94be6141-84e1-4e00-9334-910d8bbad626-kube-api-access-hf6bq" (OuterVolumeSpecName: "kube-api-access-hf6bq") pod "94be6141-84e1-4e00-9334-910d8bbad626" (UID: "94be6141-84e1-4e00-9334-910d8bbad626"). InnerVolumeSpecName "kube-api-access-hf6bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:20.220418 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.220399    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-dshm" (OuterVolumeSpecName: "dshm") pod "94be6141-84e1-4e00-9334-910d8bbad626" (UID: "94be6141-84e1-4e00-9334-910d8bbad626"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:20.233073 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.233037    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94be6141-84e1-4e00-9334-910d8bbad626" (UID: "94be6141-84e1-4e00-9334-910d8bbad626"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:20.238928 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.238893    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94be6141-84e1-4e00-9334-910d8bbad626" (UID: "94be6141-84e1-4e00-9334-910d8bbad626"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:20.318059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.317972    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94be6141-84e1-4e00-9334-910d8bbad626-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.318059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.318001    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.318059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.318011    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf6bq\" (UniqueName: \"kubernetes.io/projected/94be6141-84e1-4e00-9334-910d8bbad626-kube-api-access-hf6bq\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.318059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.318019    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.318059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.318028    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94be6141-84e1-4e00-9334-910d8bbad626-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.318059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.318035    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:20.318059 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:20.318042    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94be6141-84e1-4e00-9334-910d8bbad626-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:45:21.000336 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.000292    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz" event={"ID":"dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15","Type":"ContainerDied","Data":"980657281d307e11ae685a53ba4c4a32beb20377d09b3d83f071dfd610efc5d3"}
Sep 17 18:45:21.000336 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.000326    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"
Sep 17 18:45:21.000903 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.000378    2574 scope.go:117] "RemoveContainer" containerID="cceb8da82be046afe872567404906dc090842cff88aac94f5bb639a9d9673ce3"
Sep 17 18:45:21.003048 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.003020    2574 generic.go:356] "Generic (PLEG): container finished" podID="94be6141-84e1-4e00-9334-910d8bbad626" containerID="5151fb007a3d5454c013a7510b3a4f769fa8fc897464610742552943ddd23ecb" exitCode=0
Sep 17 18:45:21.003165 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.003094    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" event={"ID":"94be6141-84e1-4e00-9334-910d8bbad626","Type":"ContainerDied","Data":"5151fb007a3d5454c013a7510b3a4f769fa8fc897464610742552943ddd23ecb"}
Sep 17 18:45:21.003165 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.003109    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"
Sep 17 18:45:21.003165 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.003143    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q" event={"ID":"94be6141-84e1-4e00-9334-910d8bbad626","Type":"ContainerDied","Data":"d08c889881f70a97082cd03d1b9dcf74263599007ddb99a1d4c2af335eac87a5"}
Sep 17 18:45:21.011556 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.011508    2574 scope.go:117] "RemoveContainer" containerID="28018769096e2f3adcfb7228e607782f717e77cf4cdcfec092a8b973a5610fce"
Sep 17 18:45:21.020621 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.020598    2574 scope.go:117] "RemoveContainer" containerID="ae5d8cb3e85fe4391a5a012438ebfa02631cf439eeffe444ec67ff6c0a5e4fe2"
Sep 17 18:45:21.024870 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.024842    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"]
Sep 17 18:45:21.028833 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.028804    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-67szcqz"]
Sep 17 18:45:21.030802 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.030782    2574 scope.go:117] "RemoveContainer" containerID="119eee43b4e9a0ff106f1d5756a06e59e7ba25246ea1e6232feccf08d335510f"
Sep 17 18:45:21.039916 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.039893    2574 scope.go:117] "RemoveContainer" containerID="5151fb007a3d5454c013a7510b3a4f769fa8fc897464610742552943ddd23ecb"
Sep 17 18:45:21.039985 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.039894    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"]
Sep 17 18:45:21.044737 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.044710    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67855d77885xs4q"]
Sep 17 18:45:21.048939 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.048916    2574 scope.go:117] "RemoveContainer" containerID="ae5d8cb3e85fe4391a5a012438ebfa02631cf439eeffe444ec67ff6c0a5e4fe2"
Sep 17 18:45:21.049254 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:45:21.049232    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5d8cb3e85fe4391a5a012438ebfa02631cf439eeffe444ec67ff6c0a5e4fe2\": container with ID starting with ae5d8cb3e85fe4391a5a012438ebfa02631cf439eeffe444ec67ff6c0a5e4fe2 not found: ID does not exist" containerID="ae5d8cb3e85fe4391a5a012438ebfa02631cf439eeffe444ec67ff6c0a5e4fe2"
Sep 17 18:45:21.049335 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.049268    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5d8cb3e85fe4391a5a012438ebfa02631cf439eeffe444ec67ff6c0a5e4fe2"} err="failed to get container status \"ae5d8cb3e85fe4391a5a012438ebfa02631cf439eeffe444ec67ff6c0a5e4fe2\": rpc error: code = NotFound desc = could not find container \"ae5d8cb3e85fe4391a5a012438ebfa02631cf439eeffe444ec67ff6c0a5e4fe2\": container with ID starting with ae5d8cb3e85fe4391a5a012438ebfa02631cf439eeffe444ec67ff6c0a5e4fe2 not found: ID does not exist"
Sep 17 18:45:21.049335 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.049298    2574 scope.go:117] "RemoveContainer" containerID="119eee43b4e9a0ff106f1d5756a06e59e7ba25246ea1e6232feccf08d335510f"
Sep 17 18:45:21.049609 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:45:21.049590    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119eee43b4e9a0ff106f1d5756a06e59e7ba25246ea1e6232feccf08d335510f\": container with ID starting with 119eee43b4e9a0ff106f1d5756a06e59e7ba25246ea1e6232feccf08d335510f not found: ID does not exist" containerID="119eee43b4e9a0ff106f1d5756a06e59e7ba25246ea1e6232feccf08d335510f"
Sep 17 18:45:21.049655 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.049615    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119eee43b4e9a0ff106f1d5756a06e59e7ba25246ea1e6232feccf08d335510f"} err="failed to get container status \"119eee43b4e9a0ff106f1d5756a06e59e7ba25246ea1e6232feccf08d335510f\": rpc error: code = NotFound desc = could not find container \"119eee43b4e9a0ff106f1d5756a06e59e7ba25246ea1e6232feccf08d335510f\": container with ID starting with 119eee43b4e9a0ff106f1d5756a06e59e7ba25246ea1e6232feccf08d335510f not found: ID does not exist"
Sep 17 18:45:21.049655 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.049631    2574 scope.go:117] "RemoveContainer" containerID="5151fb007a3d5454c013a7510b3a4f769fa8fc897464610742552943ddd23ecb"
Sep 17 18:45:21.049893 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:45:21.049875    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5151fb007a3d5454c013a7510b3a4f769fa8fc897464610742552943ddd23ecb\": container with ID starting with 5151fb007a3d5454c013a7510b3a4f769fa8fc897464610742552943ddd23ecb not found: ID does not exist" containerID="5151fb007a3d5454c013a7510b3a4f769fa8fc897464610742552943ddd23ecb"
Sep 17 18:45:21.049943 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.049900    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5151fb007a3d5454c013a7510b3a4f769fa8fc897464610742552943ddd23ecb"} err="failed to get container status \"5151fb007a3d5454c013a7510b3a4f769fa8fc897464610742552943ddd23ecb\": rpc error: code = NotFound desc = could not find container \"5151fb007a3d5454c013a7510b3a4f769fa8fc897464610742552943ddd23ecb\": container with ID starting with 5151fb007a3d5454c013a7510b3a4f769fa8fc897464610742552943ddd23ecb not found: ID does not exist"
Sep 17 18:45:21.939104 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.939066    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94be6141-84e1-4e00-9334-910d8bbad626" path="/var/lib/kubelet/pods/94be6141-84e1-4e00-9334-910d8bbad626/volumes"
Sep 17 18:45:21.939708 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:21.939684    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" path="/var/lib/kubelet/pods/dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15/volumes"
Sep 17 18:45:54.040121 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:54.040078    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:55.348027 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:55.347977    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:57.829460 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.829370    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"]
Sep 17 18:45:57.830018 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.829996    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main"
Sep 17 18:45:57.830065 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830024    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main"
Sep 17 18:45:57.830065 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830040    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main"
Sep 17 18:45:57.830065 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830050    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main"
Sep 17 18:45:57.830166 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830065    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="storage-initializer"
Sep 17 18:45:57.830166 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830074    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="storage-initializer"
Sep 17 18:45:57.830166 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830092    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf57f11b-35a9-446a-bedf-6e4901b4aa24" containerName="main"
Sep 17 18:45:57.830166 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830101    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf57f11b-35a9-446a-bedf-6e4901b4aa24" containerName="main"
Sep 17 18:45:57.830166 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830114    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="llm-d-routing-sidecar"
Sep 17 18:45:57.830166 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830123    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="llm-d-routing-sidecar"
Sep 17 18:45:57.830166 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830134    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="storage-initializer"
Sep 17 18:45:57.830166 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830142    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="storage-initializer"
Sep 17 18:45:57.830427 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830240    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcb3dec5-1d18-4e94-bb3e-5fd1f2ae9b15" containerName="main"
Sep 17 18:45:57.830427 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830266    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="main"
Sep 17 18:45:57.830427 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830277    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf57f11b-35a9-446a-bedf-6e4901b4aa24" containerName="main"
Sep 17 18:45:57.830427 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.830289    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="94be6141-84e1-4e00-9334-910d8bbad626" containerName="llm-d-routing-sidecar"
Sep 17 18:45:57.834019 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.833986    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:57.836444 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.836413    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:45:57.836643 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.836426    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-qlz97\""
Sep 17 18:45:57.836643 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.836570    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 18:45:57.836909 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.836884    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 18:45:57.837177 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.836907    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 18:45:57.837177 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.836923    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 18:45:57.837354 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.836987    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-xfcmt\""
Sep 17 18:45:57.838064 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.838038    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"]
Sep 17 18:45:57.841581 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.841561    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:57.845981 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.845956    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"]
Sep 17 18:45:57.853618 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.853591    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"]
Sep 17 18:45:57.875372 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.875337    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:57.947786 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.947750    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vgj\" (UniqueName: \"kubernetes.io/projected/a5133254-fbf3-4c19-b318-d9a49cc43eed-kube-api-access-54vgj\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:57.947786 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.947789    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d28a47a-e828-434d-9ef2-701bf54ffda9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:57.948006 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.947811    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-dshm\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:57.948006 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.947836    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdvhb\" (UniqueName: \"kubernetes.io/projected/8d28a47a-e828-434d-9ef2-701bf54ffda9-kube-api-access-cdvhb\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:57.948006 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.947888    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:57.948006 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.947931    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:57.948006 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.947952    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:57.948006 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.947980    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5133254-fbf3-4c19-b318-d9a49cc43eed-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:57.948006 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.948000    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:57.948235 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.948055    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:57.948235 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.948077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:57.948235 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.948109    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:57.948235 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.948148    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-home\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:57.948235 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.948172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5133254-fbf3-4c19-b318-d9a49cc43eed-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:57.948235 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.948196    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d28a47a-e828-434d-9ef2-701bf54ffda9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:57.948235 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:57.948216    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.048787 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.048740    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.048787 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.048793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.049057 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.048822    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.049057 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.048860    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5133254-fbf3-4c19-b318-d9a49cc43eed-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.049057 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.048894    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.049057 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.048928    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.049057 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.048956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.049057 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.048993    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.049057 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.049024    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-home\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.049057 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.049056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5133254-fbf3-4c19-b318-d9a49cc43eed-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.049483 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.049082    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d28a47a-e828-434d-9ef2-701bf54ffda9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.049483 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.049105    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.049483 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.049151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vgj\" (UniqueName: \"kubernetes.io/projected/a5133254-fbf3-4c19-b318-d9a49cc43eed-kube-api-access-54vgj\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.049483 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.049182    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d28a47a-e828-434d-9ef2-701bf54ffda9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.049483 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.049216    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-dshm\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.049483 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.049245    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvhb\" (UniqueName: \"kubernetes.io/projected/8d28a47a-e828-434d-9ef2-701bf54ffda9-kube-api-access-cdvhb\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.049483 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.049272    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.049483 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.049245    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.049483 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.049314    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.049483 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.049339    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.050030 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.049716    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.050030 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.049757    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.050197 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.050112    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d28a47a-e828-434d-9ef2-701bf54ffda9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.050420 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.050397    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-home\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.050546 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.050445    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.050725 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.050704    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5133254-fbf3-4c19-b318-d9a49cc43eed-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.051846 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.051818    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-dshm\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.052096 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.052065    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.052096 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.052082    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5133254-fbf3-4c19-b318-d9a49cc43eed-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.052463 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.052444    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d28a47a-e828-434d-9ef2-701bf54ffda9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.057300 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.057276    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvhb\" (UniqueName: \"kubernetes.io/projected/8d28a47a-e828-434d-9ef2-701bf54ffda9-kube-api-access-cdvhb\") pod \"custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.057389 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.057361    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vgj\" (UniqueName: \"kubernetes.io/projected/a5133254-fbf3-4c19-b318-d9a49cc43eed-kube-api-access-54vgj\") pod \"custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.145896 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.145789    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:58.146734 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.146714    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:45:58.155994 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.155970    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:45:58.433915 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.433884    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"]
Sep 17 18:45:58.436356 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:58.436325    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"]
Sep 17 18:45:58.436475 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:45:58.436358    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d28a47a_e828_434d_9ef2_701bf54ffda9.slice/crio-fd5fcb972a9d0da6f3fb3d851ace0ce7ddfacfa7f14a66eaed48ba3cbea37954 WatchSource:0}: Error finding container fd5fcb972a9d0da6f3fb3d851ace0ce7ddfacfa7f14a66eaed48ba3cbea37954: Status 404 returned error can't find the container with id fd5fcb972a9d0da6f3fb3d851ace0ce7ddfacfa7f14a66eaed48ba3cbea37954
Sep 17 18:45:58.437856 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:45:58.437831    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5133254_fbf3_4c19_b318_d9a49cc43eed.slice/crio-63238d9e9c05d824c0c7fd9616177e40369cd15ab2fb046d7934d21b09673210 WatchSource:0}: Error finding container 63238d9e9c05d824c0c7fd9616177e40369cd15ab2fb046d7934d21b09673210: Status 404 returned error can't find the container with id 63238d9e9c05d824c0c7fd9616177e40369cd15ab2fb046d7934d21b09673210
Sep 17 18:45:59.211801 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:59.211757    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" event={"ID":"a5133254-fbf3-4c19-b318-d9a49cc43eed","Type":"ContainerStarted","Data":"8493e0ca8b4919bee7513322d8c858c8658e5e5e11c106da0c602aad34767703"}
Sep 17 18:45:59.212323 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:59.211829    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" event={"ID":"a5133254-fbf3-4c19-b318-d9a49cc43eed","Type":"ContainerStarted","Data":"63238d9e9c05d824c0c7fd9616177e40369cd15ab2fb046d7934d21b09673210"}
Sep 17 18:45:59.213499 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:59.213467    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" event={"ID":"8d28a47a-e828-434d-9ef2-701bf54ffda9","Type":"ContainerStarted","Data":"c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862"}
Sep 17 18:45:59.213625 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:59.213536    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" event={"ID":"8d28a47a-e828-434d-9ef2-701bf54ffda9","Type":"ContainerStarted","Data":"fd5fcb972a9d0da6f3fb3d851ace0ce7ddfacfa7f14a66eaed48ba3cbea37954"}
Sep 17 18:45:59.213625 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:45:59.213558    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:46:00.224745 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:00.224696    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" event={"ID":"8d28a47a-e828-434d-9ef2-701bf54ffda9","Type":"ContainerStarted","Data":"71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767"}
Sep 17 18:46:01.226159 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:01.226062    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:01.233907 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:01.233868    2574 generic.go:356] "Generic (PLEG): container finished" podID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerID="8493e0ca8b4919bee7513322d8c858c8658e5e5e11c106da0c602aad34767703" exitCode=0
Sep 17 18:46:01.234108 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:01.233939    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" event={"ID":"a5133254-fbf3-4c19-b318-d9a49cc43eed","Type":"ContainerDied","Data":"8493e0ca8b4919bee7513322d8c858c8658e5e5e11c106da0c602aad34767703"}
Sep 17 18:46:02.242466 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:02.242361    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" event={"ID":"a5133254-fbf3-4c19-b318-d9a49cc43eed","Type":"ContainerStarted","Data":"306fa33f075912d15b35e4659d374c9ba756e91b63753fe75d670325648e454f"}
Sep 17 18:46:02.244334 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:02.244292    2574 generic.go:356] "Generic (PLEG): container finished" podID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerID="71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767" exitCode=0
Sep 17 18:46:02.244483 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:02.244355    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" event={"ID":"8d28a47a-e828-434d-9ef2-701bf54ffda9","Type":"ContainerDied","Data":"71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767"}
Sep 17 18:46:02.267713 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:02.267662    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podStartSLOduration=2.866678781 podStartE2EDuration="5.267644507s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.400965726s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:02.265630516 +0000 UTC m=+2932.566452841" watchObservedRunningTime="2026-09-17 18:46:02.267644507 +0000 UTC m=+2932.568466830"
Sep 17 18:46:03.254925 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:03.254874    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" event={"ID":"8d28a47a-e828-434d-9ef2-701bf54ffda9","Type":"ContainerStarted","Data":"fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd"}
Sep 17 18:46:03.280931 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:03.280860    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podStartSLOduration=3.7319521719999997 podStartE2EDuration="6.280835073s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.548882901s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:03.278116481 +0000 UTC m=+2933.578938806" watchObservedRunningTime="2026-09-17 18:46:03.280835073 +0000 UTC m=+2933.581657399"
Sep 17 18:46:07.110878 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:07.110834    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:07.238574 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:07.238533    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:08.147264 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:08.147219    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:46:08.147771 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:08.147305    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:46:08.148999 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:08.148960    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:46:08.156092 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:08.156056    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:46:08.156255 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:08.156130    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:46:08.157730 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:08.157442    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:46:08.165919 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:08.165890    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:46:17.061077 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:17.061036    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:18.147397 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:18.147346    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:46:18.156630 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:18.156575    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:46:28.148021 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:28.147921    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:46:28.156698 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:28.156660    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:46:38.147562 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:38.147492    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:46:38.156838 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:38.156803    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:46:45.354078 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:45.354039    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:48.147734 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:48.147663    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:46:48.157214 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:48.157167    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:46:58.147524 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:58.147429    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:46:58.157153 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:46:58.157110    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:47:05.409364 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:05.409322    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:05.516779 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:05.516734    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:05.610715 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:05.610682    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:08.147294 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:08.147249    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:47:08.157423 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:08.157366    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:47:11.526055 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:11.526015    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:11.626930 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:11.626888    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:12.098463 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:12.098432    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:47:12.113726 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:12.113699    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:47:13.006191 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:13.006142    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:18.148035 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:18.147982    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:47:18.156325 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:18.156288    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:47:28.147385 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:28.147318    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:47:28.156353 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:28.156318    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:47:38.147583 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:38.147507    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:47:38.156896 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:38.156851    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:47:48.147998 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:48.147942    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:47:48.156847 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:48.156807    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:47:58.147649 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:58.147596    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:47:58.156492 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:47:58.156451    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:48:06.690143 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:06.690108    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:08.147974 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:08.147910    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:48:08.157283 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:08.157232    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:48:18.147207 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:18.147159    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:48:18.156416 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:18.156375    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:48:28.147554 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:28.147489    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:48:28.156719 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:28.156680    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:48:38.147899 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:38.147845    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8001/health\": dial tcp 10.134.0.78:8001: connect: connection refused"
Sep 17 18:48:38.157124 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:38.157089    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Sep 17 18:48:48.157354 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:48.157321    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:48:48.169586 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:48.169554    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:48:48.178287 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:48.178259    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:48:48.181786 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:48.181759    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:48:58.112190 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:58.112150    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:58.126678 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:58.126641    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"]
Sep 17 18:48:58.130321 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:58.130285    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"]
Sep 17 18:48:58.186766 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:48:58.186709    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:02.134042 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:02.134003    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:13.151807 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:13.151762    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main" containerID="cri-o://306fa33f075912d15b35e4659d374c9ba756e91b63753fe75d670325648e454f" gracePeriod=45
Sep 17 18:49:13.158167 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:13.158132    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main" containerID="cri-o://fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd" gracePeriod=45
Sep 17 18:49:13.985688 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:13.985656    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="llm-d-routing-sidecar" containerID="cri-o://c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862" gracePeriod=45
Sep 17 18:49:14.121924 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.121898    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:49:14.141402 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.141364    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-tmp-dir\") pod \"a5133254-fbf3-4c19-b318-d9a49cc43eed\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") "
Sep 17 18:49:14.141402 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.141399    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-model-cache\") pod \"a5133254-fbf3-4c19-b318-d9a49cc43eed\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") "
Sep 17 18:49:14.141670 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.141427    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5133254-fbf3-4c19-b318-d9a49cc43eed-tls-certs\") pod \"a5133254-fbf3-4c19-b318-d9a49cc43eed\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") "
Sep 17 18:49:14.141670 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.141453    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-kserve-provision-location\") pod \"a5133254-fbf3-4c19-b318-d9a49cc43eed\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") "
Sep 17 18:49:14.141670 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.141475    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5133254-fbf3-4c19-b318-d9a49cc43eed-cabundle-cert\") pod \"a5133254-fbf3-4c19-b318-d9a49cc43eed\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") "
Sep 17 18:49:14.141670 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.141494    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-home\") pod \"a5133254-fbf3-4c19-b318-d9a49cc43eed\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") "
Sep 17 18:49:14.141670 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.141552    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54vgj\" (UniqueName: \"kubernetes.io/projected/a5133254-fbf3-4c19-b318-d9a49cc43eed-kube-api-access-54vgj\") pod \"a5133254-fbf3-4c19-b318-d9a49cc43eed\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") "
Sep 17 18:49:14.141670 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.141568    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-dshm\") pod \"a5133254-fbf3-4c19-b318-d9a49cc43eed\" (UID: \"a5133254-fbf3-4c19-b318-d9a49cc43eed\") "
Sep 17 18:49:14.141955 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.141683    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5133254-fbf3-4c19-b318-d9a49cc43eed" (UID: "a5133254-fbf3-4c19-b318-d9a49cc43eed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:14.141955 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.141855    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.142555 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.142498    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5133254-fbf3-4c19-b318-d9a49cc43eed-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5133254-fbf3-4c19-b318-d9a49cc43eed" (UID: "a5133254-fbf3-4c19-b318-d9a49cc43eed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:49:14.142774 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.142747    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-home" (OuterVolumeSpecName: "home") pod "a5133254-fbf3-4c19-b318-d9a49cc43eed" (UID: "a5133254-fbf3-4c19-b318-d9a49cc43eed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:14.145413 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.145344    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5133254-fbf3-4c19-b318-d9a49cc43eed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5133254-fbf3-4c19-b318-d9a49cc43eed" (UID: "a5133254-fbf3-4c19-b318-d9a49cc43eed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:49:14.145413 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.145363    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5133254-fbf3-4c19-b318-d9a49cc43eed-kube-api-access-54vgj" (OuterVolumeSpecName: "kube-api-access-54vgj") pod "a5133254-fbf3-4c19-b318-d9a49cc43eed" (UID: "a5133254-fbf3-4c19-b318-d9a49cc43eed"). InnerVolumeSpecName "kube-api-access-54vgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:49:14.145616 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.145477    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-dshm" (OuterVolumeSpecName: "dshm") pod "a5133254-fbf3-4c19-b318-d9a49cc43eed" (UID: "a5133254-fbf3-4c19-b318-d9a49cc43eed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:14.161667 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.161632    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5133254-fbf3-4c19-b318-d9a49cc43eed" (UID: "a5133254-fbf3-4c19-b318-d9a49cc43eed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:14.171428 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.171394    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5133254-fbf3-4c19-b318-d9a49cc43eed" (UID: "a5133254-fbf3-4c19-b318-d9a49cc43eed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:14.240111 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.240075    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:49:14.242781 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.242757    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-dshm\") pod \"8d28a47a-e828-434d-9ef2-701bf54ffda9\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") "
Sep 17 18:49:14.242934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.242790    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-kserve-provision-location\") pod \"8d28a47a-e828-434d-9ef2-701bf54ffda9\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") "
Sep 17 18:49:14.242934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.242814    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d28a47a-e828-434d-9ef2-701bf54ffda9-cabundle-cert\") pod \"8d28a47a-e828-434d-9ef2-701bf54ffda9\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") "
Sep 17 18:49:14.242934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.242850    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdvhb\" (UniqueName: \"kubernetes.io/projected/8d28a47a-e828-434d-9ef2-701bf54ffda9-kube-api-access-cdvhb\") pod \"8d28a47a-e828-434d-9ef2-701bf54ffda9\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") "
Sep 17 18:49:14.242934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.242864    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-tmp-dir\") pod \"8d28a47a-e828-434d-9ef2-701bf54ffda9\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") "
Sep 17 18:49:14.242934 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.242911    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d28a47a-e828-434d-9ef2-701bf54ffda9-tls-certs\") pod \"8d28a47a-e828-434d-9ef2-701bf54ffda9\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") "
Sep 17 18:49:14.243192 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.242960    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-home\") pod \"8d28a47a-e828-434d-9ef2-701bf54ffda9\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") "
Sep 17 18:49:14.243192 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.242994    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-model-cache\") pod \"8d28a47a-e828-434d-9ef2-701bf54ffda9\" (UID: \"8d28a47a-e828-434d-9ef2-701bf54ffda9\") "
Sep 17 18:49:14.243302 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.243226    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.243302 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.243224    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d28a47a-e828-434d-9ef2-701bf54ffda9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d28a47a-e828-434d-9ef2-701bf54ffda9" (UID: "8d28a47a-e828-434d-9ef2-701bf54ffda9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:49:14.243302 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.243245    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5133254-fbf3-4c19-b318-d9a49cc43eed-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.243302 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.243259    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.243302 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.243272    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5133254-fbf3-4c19-b318-d9a49cc43eed-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.243302 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.243283    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.243302 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.243296    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54vgj\" (UniqueName: \"kubernetes.io/projected/a5133254-fbf3-4c19-b318-d9a49cc43eed-kube-api-access-54vgj\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.243887 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.243308    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5133254-fbf3-4c19-b318-d9a49cc43eed-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.244088 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.244062    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-home" (OuterVolumeSpecName: "home") pod "8d28a47a-e828-434d-9ef2-701bf54ffda9" (UID: "8d28a47a-e828-434d-9ef2-701bf54ffda9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:14.244463 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.244439    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d28a47a-e828-434d-9ef2-701bf54ffda9" (UID: "8d28a47a-e828-434d-9ef2-701bf54ffda9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:14.245450 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.245425    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-dshm" (OuterVolumeSpecName: "dshm") pod "8d28a47a-e828-434d-9ef2-701bf54ffda9" (UID: "8d28a47a-e828-434d-9ef2-701bf54ffda9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:14.245695 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.245673    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d28a47a-e828-434d-9ef2-701bf54ffda9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d28a47a-e828-434d-9ef2-701bf54ffda9" (UID: "8d28a47a-e828-434d-9ef2-701bf54ffda9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:49:14.245765 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.245688    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d28a47a-e828-434d-9ef2-701bf54ffda9-kube-api-access-cdvhb" (OuterVolumeSpecName: "kube-api-access-cdvhb") pod "8d28a47a-e828-434d-9ef2-701bf54ffda9" (UID: "8d28a47a-e828-434d-9ef2-701bf54ffda9"). InnerVolumeSpecName "kube-api-access-cdvhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:49:14.257973 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.257944    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d28a47a-e828-434d-9ef2-701bf54ffda9" (UID: "8d28a47a-e828-434d-9ef2-701bf54ffda9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:14.260954 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.260928    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d28a47a-e828-434d-9ef2-701bf54ffda9" (UID: "8d28a47a-e828-434d-9ef2-701bf54ffda9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:14.344508 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.344471    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d28a47a-e828-434d-9ef2-701bf54ffda9-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.344508 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.344503    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdvhb\" (UniqueName: \"kubernetes.io/projected/8d28a47a-e828-434d-9ef2-701bf54ffda9-kube-api-access-cdvhb\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.344508 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.344539    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.344771 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.344553    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d28a47a-e828-434d-9ef2-701bf54ffda9-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.344771 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.344564    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.344771 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.344575    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.344771 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.344588    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.344771 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.344596    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d28a47a-e828-434d-9ef2-701bf54ffda9-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:14.418127 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.418034    2574 generic.go:356] "Generic (PLEG): container finished" podID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerID="306fa33f075912d15b35e4659d374c9ba756e91b63753fe75d670325648e454f" exitCode=0
Sep 17 18:49:14.418127 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.418087    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"
Sep 17 18:49:14.418381 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.418120    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" event={"ID":"a5133254-fbf3-4c19-b318-d9a49cc43eed","Type":"ContainerDied","Data":"306fa33f075912d15b35e4659d374c9ba756e91b63753fe75d670325648e454f"}
Sep 17 18:49:14.418381 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.418187    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6" event={"ID":"a5133254-fbf3-4c19-b318-d9a49cc43eed","Type":"ContainerDied","Data":"63238d9e9c05d824c0c7fd9616177e40369cd15ab2fb046d7934d21b09673210"}
Sep 17 18:49:14.418381 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.418215    2574 scope.go:117] "RemoveContainer" containerID="306fa33f075912d15b35e4659d374c9ba756e91b63753fe75d670325648e454f"
Sep 17 18:49:14.420401 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.420373    2574 generic.go:356] "Generic (PLEG): container finished" podID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerID="fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd" exitCode=0
Sep 17 18:49:14.420401 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.420399    2574 generic.go:356] "Generic (PLEG): container finished" podID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerID="c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862" exitCode=0
Sep 17 18:49:14.420625 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.420424    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"
Sep 17 18:49:14.420625 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.420447    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" event={"ID":"8d28a47a-e828-434d-9ef2-701bf54ffda9","Type":"ContainerDied","Data":"fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd"}
Sep 17 18:49:14.420625 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.420497    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" event={"ID":"8d28a47a-e828-434d-9ef2-701bf54ffda9","Type":"ContainerDied","Data":"c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862"}
Sep 17 18:49:14.420625 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.420539    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6" event={"ID":"8d28a47a-e828-434d-9ef2-701bf54ffda9","Type":"ContainerDied","Data":"fd5fcb972a9d0da6f3fb3d851ace0ce7ddfacfa7f14a66eaed48ba3cbea37954"}
Sep 17 18:49:14.429264 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.429246    2574 scope.go:117] "RemoveContainer" containerID="8493e0ca8b4919bee7513322d8c858c8658e5e5e11c106da0c602aad34767703"
Sep 17 18:49:14.438304 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.438284    2574 scope.go:117] "RemoveContainer" containerID="306fa33f075912d15b35e4659d374c9ba756e91b63753fe75d670325648e454f"
Sep 17 18:49:14.438597 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:49:14.438574    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306fa33f075912d15b35e4659d374c9ba756e91b63753fe75d670325648e454f\": container with ID starting with 306fa33f075912d15b35e4659d374c9ba756e91b63753fe75d670325648e454f not found: ID does not exist" containerID="306fa33f075912d15b35e4659d374c9ba756e91b63753fe75d670325648e454f"
Sep 17 18:49:14.438668 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.438605    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306fa33f075912d15b35e4659d374c9ba756e91b63753fe75d670325648e454f"} err="failed to get container status \"306fa33f075912d15b35e4659d374c9ba756e91b63753fe75d670325648e454f\": rpc error: code = NotFound desc = could not find container \"306fa33f075912d15b35e4659d374c9ba756e91b63753fe75d670325648e454f\": container with ID starting with 306fa33f075912d15b35e4659d374c9ba756e91b63753fe75d670325648e454f not found: ID does not exist"
Sep 17 18:49:14.438668 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.438621    2574 scope.go:117] "RemoveContainer" containerID="8493e0ca8b4919bee7513322d8c858c8658e5e5e11c106da0c602aad34767703"
Sep 17 18:49:14.438912 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:49:14.438895    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8493e0ca8b4919bee7513322d8c858c8658e5e5e11c106da0c602aad34767703\": container with ID starting with 8493e0ca8b4919bee7513322d8c858c8658e5e5e11c106da0c602aad34767703 not found: ID does not exist" containerID="8493e0ca8b4919bee7513322d8c858c8658e5e5e11c106da0c602aad34767703"
Sep 17 18:49:14.438966 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.438918    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8493e0ca8b4919bee7513322d8c858c8658e5e5e11c106da0c602aad34767703"} err="failed to get container status \"8493e0ca8b4919bee7513322d8c858c8658e5e5e11c106da0c602aad34767703\": rpc error: code = NotFound desc = could not find container \"8493e0ca8b4919bee7513322d8c858c8658e5e5e11c106da0c602aad34767703\": container with ID starting with 8493e0ca8b4919bee7513322d8c858c8658e5e5e11c106da0c602aad34767703 not found: ID does not exist"
Sep 17 18:49:14.438966 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.438934    2574 scope.go:117] "RemoveContainer" containerID="fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd"
Sep 17 18:49:14.446091 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.446064    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"]
Sep 17 18:49:14.447946 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.447928    2574 scope.go:117] "RemoveContainer" containerID="71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767"
Sep 17 18:49:14.449760 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.449737    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-846fb75b69-q7pn6"]
Sep 17 18:49:14.456737 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.456715    2574 scope.go:117] "RemoveContainer" containerID="c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862"
Sep 17 18:49:14.461021 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.460999    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"]
Sep 17 18:49:14.465815 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.465791    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6f5c57965f-gx8c6"]
Sep 17 18:49:14.466384 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.466367    2574 scope.go:117] "RemoveContainer" containerID="fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd"
Sep 17 18:49:14.466704 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:49:14.466683    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd\": container with ID starting with fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd not found: ID does not exist" containerID="fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd"
Sep 17 18:49:14.466780 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.466715    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd"} err="failed to get container status \"fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd\": rpc error: code = NotFound desc = could not find container \"fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd\": container with ID starting with fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd not found: ID does not exist"
Sep 17 18:49:14.466780 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.466742    2574 scope.go:117] "RemoveContainer" containerID="71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767"
Sep 17 18:49:14.467012 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:49:14.466992    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767\": container with ID starting with 71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767 not found: ID does not exist" containerID="71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767"
Sep 17 18:49:14.467056 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.467018    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767"} err="failed to get container status \"71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767\": rpc error: code = NotFound desc = could not find container \"71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767\": container with ID starting with 71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767 not found: ID does not exist"
Sep 17 18:49:14.467056 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.467032    2574 scope.go:117] "RemoveContainer" containerID="c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862"
Sep 17 18:49:14.467291 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:49:14.467273    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862\": container with ID starting with c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862 not found: ID does not exist" containerID="c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862"
Sep 17 18:49:14.467335 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.467297    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862"} err="failed to get container status \"c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862\": rpc error: code = NotFound desc = could not find container \"c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862\": container with ID starting with c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862 not found: ID does not exist"
Sep 17 18:49:14.467335 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.467318    2574 scope.go:117] "RemoveContainer" containerID="fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd"
Sep 17 18:49:14.467575 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.467555    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd"} err="failed to get container status \"fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd\": rpc error: code = NotFound desc = could not find container \"fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd\": container with ID starting with fd743731203bda1e32912fdea306003a6451cb702606830c24146d2f9a13c1cd not found: ID does not exist"
Sep 17 18:49:14.467654 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.467576    2574 scope.go:117] "RemoveContainer" containerID="71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767"
Sep 17 18:49:14.467836 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.467820    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767"} err="failed to get container status \"71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767\": rpc error: code = NotFound desc = could not find container \"71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767\": container with ID starting with 71df212bb596da5f30f36cb6fec9ac58633b4c0a48a0a95119efd04c773f9767 not found: ID does not exist"
Sep 17 18:49:14.467898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.467837    2574 scope.go:117] "RemoveContainer" containerID="c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862"
Sep 17 18:49:14.468058 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:14.468037    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862"} err="failed to get container status \"c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862\": rpc error: code = NotFound desc = could not find container \"c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862\": container with ID starting with c02734707404b1166fb968a30530919bf5c9fa9b86c2543954506ef128732862 not found: ID does not exist"
Sep 17 18:49:15.938395 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:15.938359    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" path="/var/lib/kubelet/pods/8d28a47a-e828-434d-9ef2-701bf54ffda9/volumes"
Sep 17 18:49:15.938931 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:15.938916    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" path="/var/lib/kubelet/pods/a5133254-fbf3-4c19-b318-d9a49cc43eed/volumes"
Sep 17 18:49:22.223290 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:22.223249    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:22.328776 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:22.328735    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:22.426878 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:22.426845    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:22.525966 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:22.525931    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:22.627645 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:22.627604    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:24.914984 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:24.914946    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:24.972087 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:24.972049    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:25.542239 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.542199    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"]
Sep 17 18:49:25.542616 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.542602    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="llm-d-routing-sidecar"
Sep 17 18:49:25.542667 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.542618    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="llm-d-routing-sidecar"
Sep 17 18:49:25.542667 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.542628    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main"
Sep 17 18:49:25.542667 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.542633    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main"
Sep 17 18:49:25.542667 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.542645    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="storage-initializer"
Sep 17 18:49:25.542667 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.542651    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="storage-initializer"
Sep 17 18:49:25.542667 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.542662    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="storage-initializer"
Sep 17 18:49:25.542667 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.542668    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="storage-initializer"
Sep 17 18:49:25.542903 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.542678    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main"
Sep 17 18:49:25.542903 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.542683    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main"
Sep 17 18:49:25.542903 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.542737    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="llm-d-routing-sidecar"
Sep 17 18:49:25.542903 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.542743    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5133254-fbf3-4c19-b318-d9a49cc43eed" containerName="main"
Sep 17 18:49:25.542903 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.542748    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d28a47a-e828-434d-9ef2-701bf54ffda9" containerName="main"
Sep 17 18:49:25.547639 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.547618    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"
Sep 17 18:49:25.550148 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.550117    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 18:49:25.550314 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.550149    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 18:49:25.550650 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.550621    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 18:49:25.550781 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.550654    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-cjt8p\""
Sep 17 18:49:25.553120 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.553095    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"]
Sep 17 18:49:25.645262 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.645220    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc8w8\" (UniqueName: \"kubernetes.io/projected/f6b30beb-fe9d-4bfc-9e43-081def206fa9-kube-api-access-lc8w8\") pod \"e2e-pvc-lora-download-5mzgz\" (UID: \"f6b30beb-fe9d-4bfc-9e43-081def206fa9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"
Sep 17 18:49:25.677495 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.677460    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:25.746555 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.746489    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc8w8\" (UniqueName: \"kubernetes.io/projected/f6b30beb-fe9d-4bfc-9e43-081def206fa9-kube-api-access-lc8w8\") pod \"e2e-pvc-lora-download-5mzgz\" (UID: \"f6b30beb-fe9d-4bfc-9e43-081def206fa9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"
Sep 17 18:49:25.754903 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:25.754874    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc8w8\" (UniqueName: \"kubernetes.io/projected/f6b30beb-fe9d-4bfc-9e43-081def206fa9-kube-api-access-lc8w8\") pod \"e2e-pvc-lora-download-5mzgz\" (UID: \"f6b30beb-fe9d-4bfc-9e43-081def206fa9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"
Sep 17 18:49:27.966639 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:27.966599    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"e2e-pvc-lora-download-5mzgz\" (UID: \"f6b30beb-fe9d-4bfc-9e43-081def206fa9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"
Sep 17 18:49:27.973987 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:27.973955    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"e2e-pvc-lora-download-5mzgz\" (UID: \"f6b30beb-fe9d-4bfc-9e43-081def206fa9\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"
Sep 17 18:49:28.067634 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:28.067599    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"e2e-pvc-lora-download-5mzgz\" (UID: \"f6b30beb-fe9d-4bfc-9e43-081def206fa9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"
Sep 17 18:49:28.067826 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:28.067710    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"e2e-pvc-lora-download-5mzgz\" (UID: \"f6b30beb-fe9d-4bfc-9e43-081def206fa9\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"
Sep 17 18:49:28.070165 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:28.070139    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"e2e-pvc-lora-download-5mzgz\" (UID: \"f6b30beb-fe9d-4bfc-9e43-081def206fa9\") DevicePath \"csi-ed995ece8c6dd2a8e1c7c5c0a4926b43ee80d8d41833eeb850296e8928cec2c8\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"
Sep 17 18:49:28.535438 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:28.535382    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"e2e-pvc-lora-download-5mzgz\" (UID: \"f6b30beb-fe9d-4bfc-9e43-081def206fa9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/c4dcef12ca6d3a2683a4d02beeee8c4c81215403052c3b6774a521b194566dcc/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"
Sep 17 18:49:28.549657 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:28.549627    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"e2e-pvc-lora-download-5mzgz\" (UID: \"f6b30beb-fe9d-4bfc-9e43-081def206fa9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"
Sep 17 18:49:28.558628 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:28.558601    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"
Sep 17 18:49:28.566806 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:28.566779    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:28.631303 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:28.631262    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:28.866290 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:28.866262    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"]
Sep 17 18:49:28.868301 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:49:28.868258    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6b30beb_fe9d_4bfc_9e43_081def206fa9.slice/crio-6dbd455cad68f83252f5fd17732823e1cffd1e3e6227f32be8afed9ce06a879a WatchSource:0}: Error finding container 6dbd455cad68f83252f5fd17732823e1cffd1e3e6227f32be8afed9ce06a879a: Status 404 returned error can't find the container with id 6dbd455cad68f83252f5fd17732823e1cffd1e3e6227f32be8afed9ce06a879a
Sep 17 18:49:29.507485 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:29.507435    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz" event={"ID":"f6b30beb-fe9d-4bfc-9e43-081def206fa9","Type":"ContainerStarted","Data":"c4fa267475e89290e9982a3c2ef4ff7e57ae88f10598c7ae9d801bcdd3bcbeb2"}
Sep 17 18:49:29.508015 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:29.507493    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz" event={"ID":"f6b30beb-fe9d-4bfc-9e43-081def206fa9","Type":"ContainerStarted","Data":"6dbd455cad68f83252f5fd17732823e1cffd1e3e6227f32be8afed9ce06a879a"}
Sep 17 18:49:29.522610 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:29.522556    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz" podStartSLOduration=7.52254031 podStartE2EDuration="7.52254031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:49:29.521694775 +0000 UTC m=+3139.822517099" watchObservedRunningTime="2026-09-17 18:49:29.52254031 +0000 UTC m=+3139.823362629"
Sep 17 18:49:31.519299 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:31.519262    2574 generic.go:356] "Generic (PLEG): container finished" podID="f6b30beb-fe9d-4bfc-9e43-081def206fa9" containerID="c4fa267475e89290e9982a3c2ef4ff7e57ae88f10598c7ae9d801bcdd3bcbeb2" exitCode=0
Sep 17 18:49:31.519299 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:31.519305    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz" event={"ID":"f6b30beb-fe9d-4bfc-9e43-081def206fa9","Type":"ContainerDied","Data":"c4fa267475e89290e9982a3c2ef4ff7e57ae88f10598c7ae9d801bcdd3bcbeb2"}
Sep 17 18:49:32.672834 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:32.672801    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"
Sep 17 18:49:32.711202 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:32.711168    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"f6b30beb-fe9d-4bfc-9e43-081def206fa9\" (UID: \"f6b30beb-fe9d-4bfc-9e43-081def206fa9\") "
Sep 17 18:49:32.711202 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:32.711209    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc8w8\" (UniqueName: \"kubernetes.io/projected/f6b30beb-fe9d-4bfc-9e43-081def206fa9-kube-api-access-lc8w8\") pod \"f6b30beb-fe9d-4bfc-9e43-081def206fa9\" (UID: \"f6b30beb-fe9d-4bfc-9e43-081def206fa9\") "
Sep 17 18:49:32.713821 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:32.713791    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b30beb-fe9d-4bfc-9e43-081def206fa9-kube-api-access-lc8w8" (OuterVolumeSpecName: "kube-api-access-lc8w8") pod "f6b30beb-fe9d-4bfc-9e43-081def206fa9" (UID: "f6b30beb-fe9d-4bfc-9e43-081def206fa9"). InnerVolumeSpecName "kube-api-access-lc8w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:49:32.715347 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:32.715323    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc" (OuterVolumeSpecName: "model-storage") pod "f6b30beb-fe9d-4bfc-9e43-081def206fa9" (UID: "f6b30beb-fe9d-4bfc-9e43-081def206fa9"). InnerVolumeSpecName "pvc-781009b5-a2c7-4181-b106-aa18d6760848". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:49:32.812363 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:32.812264    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") on node \"ip-10-0-132-0.ec2.internal\" "
Sep 17 18:49:32.812363 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:32.812303    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc8w8\" (UniqueName: \"kubernetes.io/projected/f6b30beb-fe9d-4bfc-9e43-081def206fa9-kube-api-access-lc8w8\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:49:33.530851 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:33.530817    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"
Sep 17 18:49:33.530851 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:33.530832    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz" event={"ID":"f6b30beb-fe9d-4bfc-9e43-081def206fa9","Type":"ContainerDied","Data":"6dbd455cad68f83252f5fd17732823e1cffd1e3e6227f32be8afed9ce06a879a"}
Sep 17 18:49:33.531074 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:33.530867    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dbd455cad68f83252f5fd17732823e1cffd1e3e6227f32be8afed9ce06a879a"
Sep 17 18:49:33.787290 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:33.787265    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-781009b5-a2c7-4181-b106-aa18d6760848" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc") on node "ip-10-0-132-0.ec2.internal"
Sep 17 18:49:33.821273 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:33.821236    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"csi-ed995ece8c6dd2a8e1c7c5c0a4926b43ee80d8d41833eeb850296e8928cec2c8\""
Sep 17 18:49:47.579202 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.579165    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"]
Sep 17 18:49:47.579702 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.579574    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b30beb-fe9d-4bfc-9e43-081def206fa9" containerName="storage-initializer"
Sep 17 18:49:47.579702 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.579594    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b30beb-fe9d-4bfc-9e43-081def206fa9" containerName="storage-initializer"
Sep 17 18:49:47.579780 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.579745    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6b30beb-fe9d-4bfc-9e43-081def206fa9" containerName="storage-initializer"
Sep 17 18:49:47.587439 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.587417    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.589714 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.589686    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 18:49:47.589959 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.589940    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 17 18:49:47.590062 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.590033    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 18:49:47.590138 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.590058    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 18:49:47.590138 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.590131    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:49:47.590261 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.590149    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-cjt8p\""
Sep 17 18:49:47.595768 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.595744    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"]
Sep 17 18:49:47.600282 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.600258    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:47.641175 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.641139    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f640c92c-8e6c-4fa3-8147-2cf1951e370a-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.641341 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.641184    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f640c92c-8e6c-4fa3-8147-2cf1951e370a-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.641341 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.641218    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.641341 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.641243    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.641341 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.641265    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2nrs\" (UniqueName: \"kubernetes.io/projected/f640c92c-8e6c-4fa3-8147-2cf1951e370a-kube-api-access-x2nrs\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.641341 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.641288    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.641623 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.641387    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-home\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.641623 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.641436    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-dshm\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.742056 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.742006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f640c92c-8e6c-4fa3-8147-2cf1951e370a-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.742056 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.742067    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f640c92c-8e6c-4fa3-8147-2cf1951e370a-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.742357 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.742095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.742357 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.742116    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.742357 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.742138    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2nrs\" (UniqueName: \"kubernetes.io/projected/f640c92c-8e6c-4fa3-8147-2cf1951e370a-kube-api-access-x2nrs\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.742357 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.742155    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.742357 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.742185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-home\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.742357 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.742223    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-dshm\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.742747 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.742716    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-home\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.742879 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.742763    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.742879 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.742799    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.742879 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.742855    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.743225 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.742954    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f640c92c-8e6c-4fa3-8147-2cf1951e370a-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.744839 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.744818    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-dshm\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.745002 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.744985    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f640c92c-8e6c-4fa3-8147-2cf1951e370a-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:47.750470 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:47.750445    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2nrs\" (UniqueName: \"kubernetes.io/projected/f640c92c-8e6c-4fa3-8147-2cf1951e370a-kube-api-access-x2nrs\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:48.255347 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:48.255310    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:48.354940 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:48.354908    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:48.488119 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:48.488088    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:48.549933 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:48.549853    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:52.235247 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:52.235211    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:55.018778 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:55.018739    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:57.527395 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:57.527351    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:57.532711 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:57.532685    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:57.628247 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:57.628198    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:57.628456 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:57.628355    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:57.631490 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:57.631469    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") DevicePath \"csi-ed995ece8c6dd2a8e1c7c5c0a4926b43ee80d8d41833eeb850296e8928cec2c8\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:57.756625 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:57.756593    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/c4dcef12ca6d3a2683a4d02beeee8c4c81215403052c3b6774a521b194566dcc/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:57.776568 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:57.776535    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:57.800854 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:57.800780    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:49:58.082764 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:58.082731    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"]
Sep 17 18:49:58.084129 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:49:58.084102    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf640c92c_8e6c_4fa3_8147_2cf1951e370a.slice/crio-94c8f289ecab6d1981d80dde55d94f8189e76c0384a5f1fcded6081c2db3b5d7 WatchSource:0}: Error finding container 94c8f289ecab6d1981d80dde55d94f8189e76c0384a5f1fcded6081c2db3b5d7: Status 404 returned error can't find the container with id 94c8f289ecab6d1981d80dde55d94f8189e76c0384a5f1fcded6081c2db3b5d7
Sep 17 18:49:58.662602 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:58.662565    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" event={"ID":"f640c92c-8e6c-4fa3-8147-2cf1951e370a","Type":"ContainerStarted","Data":"0d6b63dc1704d1b39968a3ac11346661e54a015c39ad00dcd32d7bf6c251e585"}
Sep 17 18:49:58.662984 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:49:58.662625    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" event={"ID":"f640c92c-8e6c-4fa3-8147-2cf1951e370a","Type":"ContainerStarted","Data":"94c8f289ecab6d1981d80dde55d94f8189e76c0384a5f1fcded6081c2db3b5d7"}
Sep 17 18:50:00.674348 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:00.674243    2574 generic.go:356] "Generic (PLEG): container finished" podID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerID="0d6b63dc1704d1b39968a3ac11346661e54a015c39ad00dcd32d7bf6c251e585" exitCode=0
Sep 17 18:50:00.674348 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:00.674318    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" event={"ID":"f640c92c-8e6c-4fa3-8147-2cf1951e370a","Type":"ContainerDied","Data":"0d6b63dc1704d1b39968a3ac11346661e54a015c39ad00dcd32d7bf6c251e585"}
Sep 17 18:50:00.806216 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:00.806172    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:00.905999 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:00.905959    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:01.682421 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:01.682375    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" event={"ID":"f640c92c-8e6c-4fa3-8147-2cf1951e370a","Type":"ContainerStarted","Data":"fd162941bf572a64b72372ececd27b99651a0ff13abe3c84dc716e3734f48231"}
Sep 17 18:50:01.706353 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:01.706295    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" podStartSLOduration=12.498817642 podStartE2EDuration="14.706274338s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.207456696s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:01.703886042 +0000 UTC m=+3172.004708390" watchObservedRunningTime="2026-09-17 18:50:01.706274338 +0000 UTC m=+3172.007096662"
Sep 17 18:50:01.801794 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:01.801750    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:50:07.801966 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:07.801922    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:50:07.803750 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:07.803710    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 18:50:17.801760 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:17.801715    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 18:50:27.801776 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:27.801720    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 18:50:37.801291 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:37.801243    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 18:50:42.038408 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:42.038361    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:45.602643 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.602602    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"]
Sep 17 18:50:45.606720 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.606689    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.609383 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.609356    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 18:50:45.609566 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.609491    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 18:50:45.609566 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.609475    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 18:50:45.609715 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.609593    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 18:50:45.609857 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.609838    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-sp9wg\""
Sep 17 18:50:45.609992 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.609840    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:50:45.618088 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.618061    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"]
Sep 17 18:50:45.640440 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.640410    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:45.678106 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.678051    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616d1c70-058d-4d94-8106-ae939ecba97d-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.678106 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.678111    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s5qx\" (UniqueName: \"kubernetes.io/projected/616d1c70-058d-4d94-8106-ae939ecba97d-kube-api-access-6s5qx\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.678397 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.678202    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.678397 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.678242    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.678397 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.678317    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/616d1c70-058d-4d94-8106-ae939ecba97d-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.678397 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.678350    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.678675 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.678424    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-home\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.678675 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.678452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.779174 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.779105    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.779174 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.779177    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-home\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.779456 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.779202    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.779456 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.779244    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616d1c70-058d-4d94-8106-ae939ecba97d-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.779456 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.779274    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s5qx\" (UniqueName: \"kubernetes.io/projected/616d1c70-058d-4d94-8106-ae939ecba97d-kube-api-access-6s5qx\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.779456 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.779339    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.779456 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.779385    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.779456 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.779451    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/616d1c70-058d-4d94-8106-ae939ecba97d-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.779859 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.779637    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.779859 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.779674    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-home\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.779965 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.779899    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.780035 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.779977    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.780216 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.780197    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/616d1c70-058d-4d94-8106-ae939ecba97d-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.781862 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.781841    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.782273 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.782237    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616d1c70-058d-4d94-8106-ae939ecba97d-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.787815 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.787783    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s5qx\" (UniqueName: \"kubernetes.io/projected/616d1c70-058d-4d94-8106-ae939ecba97d-kube-api-access-6s5qx\") pod \"router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:45.920204 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:45.920096    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:46.203710 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:46.203663    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"]
Sep 17 18:50:46.205857 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:50:46.205813    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616d1c70_058d_4d94_8106_ae939ecba97d.slice/crio-9e6e5e3ef9dd57cbe7c35d39deed97f90b7b8b2084cc19d14d45a493604c5f7e WatchSource:0}: Error finding container 9e6e5e3ef9dd57cbe7c35d39deed97f90b7b8b2084cc19d14d45a493604c5f7e: Status 404 returned error can't find the container with id 9e6e5e3ef9dd57cbe7c35d39deed97f90b7b8b2084cc19d14d45a493604c5f7e
Sep 17 18:50:46.254239 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:46.254209    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:46.360194 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:46.360155    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:46.948750 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:46.948699    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" event={"ID":"616d1c70-058d-4d94-8106-ae939ecba97d","Type":"ContainerStarted","Data":"09c3ae528060d14ec788b17e88c2ea4144eb25b80756b52941a2e60cedd983ee"}
Sep 17 18:50:46.949214 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:46.948817    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" event={"ID":"616d1c70-058d-4d94-8106-ae939ecba97d","Type":"ContainerStarted","Data":"9e6e5e3ef9dd57cbe7c35d39deed97f90b7b8b2084cc19d14d45a493604c5f7e"}
Sep 17 18:50:47.801946 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:47.801887    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 18:50:48.962457 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:48.962356    2574 generic.go:356] "Generic (PLEG): container finished" podID="616d1c70-058d-4d94-8106-ae939ecba97d" containerID="09c3ae528060d14ec788b17e88c2ea4144eb25b80756b52941a2e60cedd983ee" exitCode=0
Sep 17 18:50:48.962889 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:48.962450    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" event={"ID":"616d1c70-058d-4d94-8106-ae939ecba97d","Type":"ContainerDied","Data":"09c3ae528060d14ec788b17e88c2ea4144eb25b80756b52941a2e60cedd983ee"}
Sep 17 18:50:49.972013 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:49.971969    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" event={"ID":"616d1c70-058d-4d94-8106-ae939ecba97d","Type":"ContainerStarted","Data":"ff3b26037a00d530a11c7bd6cac2735bc8419a4e44cc417f62ad490f24a25993"}
Sep 17 18:50:49.995721 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:49.995662    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" podStartSLOduration=2.603634115 podStartE2EDuration="4.995646727s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.392012612s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:49.99416721 +0000 UTC m=+3220.294989572" watchObservedRunningTime="2026-09-17 18:50:49.995646727 +0000 UTC m=+3220.296469051"
Sep 17 18:50:55.920596 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:55.920541    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:55.923205 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:55.920619    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:50:55.923205 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:55.922434    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 18:50:57.801728 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:50:57.801667    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 18:51:05.920911 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:05.920863    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 18:51:07.801674 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:07.801624    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 18:51:15.921373 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:15.921316    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 18:51:17.811184 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:17.811145    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:51:17.818977 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:17.818949    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:51:25.921284 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:25.921236    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 18:51:35.921305 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:35.921246    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 18:51:43.556309 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:43.556261    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42_f640c92c-8e6c-4fa3-8147-2cf1951e370a/storage-initializer/0.log"
Sep 17 18:51:43.565839 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:43.565818    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42_f640c92c-8e6c-4fa3-8147-2cf1951e370a/main/0.log"
Sep 17 18:51:44.576859 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:44.576827    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"]
Sep 17 18:51:44.593098 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:44.593063    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:45.920887 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:45.920833    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 18:51:46.001011 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:46.000973    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:55.921084 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:55.921037    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 18:51:59.601226 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:59.601145    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerName="main" containerID="cri-o://fd162941bf572a64b72372ececd27b99651a0ff13abe3c84dc716e3734f48231" gracePeriod=45
Sep 17 18:51:59.812237 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:51:59.812198    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 18:52:00.380989 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.380964    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:52:00.418134 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.418100    2574 generic.go:356] "Generic (PLEG): container finished" podID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerID="fd162941bf572a64b72372ececd27b99651a0ff13abe3c84dc716e3734f48231" exitCode=0
Sep 17 18:52:00.418285 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.418144    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" event={"ID":"f640c92c-8e6c-4fa3-8147-2cf1951e370a","Type":"ContainerDied","Data":"fd162941bf572a64b72372ececd27b99651a0ff13abe3c84dc716e3734f48231"}
Sep 17 18:52:00.418285 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.418166    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"
Sep 17 18:52:00.418285 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.418196    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42" event={"ID":"f640c92c-8e6c-4fa3-8147-2cf1951e370a","Type":"ContainerDied","Data":"94c8f289ecab6d1981d80dde55d94f8189e76c0384a5f1fcded6081c2db3b5d7"}
Sep 17 18:52:00.418285 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.418226    2574 scope.go:117] "RemoveContainer" containerID="fd162941bf572a64b72372ececd27b99651a0ff13abe3c84dc716e3734f48231"
Sep 17 18:52:00.428926 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.428903    2574 scope.go:117] "RemoveContainer" containerID="0d6b63dc1704d1b39968a3ac11346661e54a015c39ad00dcd32d7bf6c251e585"
Sep 17 18:52:00.439144 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.439119    2574 scope.go:117] "RemoveContainer" containerID="fd162941bf572a64b72372ececd27b99651a0ff13abe3c84dc716e3734f48231"
Sep 17 18:52:00.439453 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:52:00.439431    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd162941bf572a64b72372ececd27b99651a0ff13abe3c84dc716e3734f48231\": container with ID starting with fd162941bf572a64b72372ececd27b99651a0ff13abe3c84dc716e3734f48231 not found: ID does not exist" containerID="fd162941bf572a64b72372ececd27b99651a0ff13abe3c84dc716e3734f48231"
Sep 17 18:52:00.439548 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.439462    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd162941bf572a64b72372ececd27b99651a0ff13abe3c84dc716e3734f48231"} err="failed to get container status \"fd162941bf572a64b72372ececd27b99651a0ff13abe3c84dc716e3734f48231\": rpc error: code = NotFound desc = could not find container \"fd162941bf572a64b72372ececd27b99651a0ff13abe3c84dc716e3734f48231\": container with ID starting with fd162941bf572a64b72372ececd27b99651a0ff13abe3c84dc716e3734f48231 not found: ID does not exist"
Sep 17 18:52:00.439548 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.439479    2574 scope.go:117] "RemoveContainer" containerID="0d6b63dc1704d1b39968a3ac11346661e54a015c39ad00dcd32d7bf6c251e585"
Sep 17 18:52:00.439748 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:52:00.439725    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6b63dc1704d1b39968a3ac11346661e54a015c39ad00dcd32d7bf6c251e585\": container with ID starting with 0d6b63dc1704d1b39968a3ac11346661e54a015c39ad00dcd32d7bf6c251e585 not found: ID does not exist" containerID="0d6b63dc1704d1b39968a3ac11346661e54a015c39ad00dcd32d7bf6c251e585"
Sep 17 18:52:00.439790 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.439758    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6b63dc1704d1b39968a3ac11346661e54a015c39ad00dcd32d7bf6c251e585"} err="failed to get container status \"0d6b63dc1704d1b39968a3ac11346661e54a015c39ad00dcd32d7bf6c251e585\": rpc error: code = NotFound desc = could not find container \"0d6b63dc1704d1b39968a3ac11346661e54a015c39ad00dcd32d7bf6c251e585\": container with ID starting with 0d6b63dc1704d1b39968a3ac11346661e54a015c39ad00dcd32d7bf6c251e585 not found: ID does not exist"
Sep 17 18:52:00.465199 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.465170    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-home\") pod \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") "
Sep 17 18:52:00.465312 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.465247    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f640c92c-8e6c-4fa3-8147-2cf1951e370a-cabundle-cert\") pod \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") "
Sep 17 18:52:00.465312 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.465296    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-model-cache\") pod \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") "
Sep 17 18:52:00.465434 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.465333    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-kserve-provision-location\") pod \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") "
Sep 17 18:52:00.466558 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.465856    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-home" (OuterVolumeSpecName: "home") pod "f640c92c-8e6c-4fa3-8147-2cf1951e370a" (UID: "f640c92c-8e6c-4fa3-8147-2cf1951e370a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:00.466558 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.466001    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f640c92c-8e6c-4fa3-8147-2cf1951e370a-tls-certs\") pod \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") "
Sep 17 18:52:00.466558 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.466055    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f640c92c-8e6c-4fa3-8147-2cf1951e370a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f640c92c-8e6c-4fa3-8147-2cf1951e370a" (UID: "f640c92c-8e6c-4fa3-8147-2cf1951e370a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:52:00.466558 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.466070    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-tmp-dir\") pod \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") "
Sep 17 18:52:00.466558 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.466181    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") pod \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") "
Sep 17 18:52:00.466558 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.466361    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-model-cache" (OuterVolumeSpecName: "model-cache") pod "f640c92c-8e6c-4fa3-8147-2cf1951e370a" (UID: "f640c92c-8e6c-4fa3-8147-2cf1951e370a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:00.472241 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.469719    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f640c92c-8e6c-4fa3-8147-2cf1951e370a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f640c92c-8e6c-4fa3-8147-2cf1951e370a" (UID: "f640c92c-8e6c-4fa3-8147-2cf1951e370a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:52:00.476822 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.476793    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "f640c92c-8e6c-4fa3-8147-2cf1951e370a" (UID: "f640c92c-8e6c-4fa3-8147-2cf1951e370a"). InnerVolumeSpecName "pvc-781009b5-a2c7-4181-b106-aa18d6760848". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:52:00.476933 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.476869    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2nrs\" (UniqueName: \"kubernetes.io/projected/f640c92c-8e6c-4fa3-8147-2cf1951e370a-kube-api-access-x2nrs\") pod \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") "
Sep 17 18:52:00.476933 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.476913    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-dshm\") pod \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\" (UID: \"f640c92c-8e6c-4fa3-8147-2cf1951e370a\") "
Sep 17 18:52:00.477420 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.477407    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f640c92c-8e6c-4fa3-8147-2cf1951e370a-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:52:00.477479 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.477445    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") on node \"ip-10-0-132-0.ec2.internal\" "
Sep 17 18:52:00.477479 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.477459    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:52:00.477624 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.477485    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f640c92c-8e6c-4fa3-8147-2cf1951e370a-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:52:00.477624 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.477497    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:52:00.484143 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.484095    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f640c92c-8e6c-4fa3-8147-2cf1951e370a" (UID: "f640c92c-8e6c-4fa3-8147-2cf1951e370a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:00.485765 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.485735    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f640c92c-8e6c-4fa3-8147-2cf1951e370a-kube-api-access-x2nrs" (OuterVolumeSpecName: "kube-api-access-x2nrs") pod "f640c92c-8e6c-4fa3-8147-2cf1951e370a" (UID: "f640c92c-8e6c-4fa3-8147-2cf1951e370a"). InnerVolumeSpecName "kube-api-access-x2nrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:52:00.489652 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.489419    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-dshm" (OuterVolumeSpecName: "dshm") pod "f640c92c-8e6c-4fa3-8147-2cf1951e370a" (UID: "f640c92c-8e6c-4fa3-8147-2cf1951e370a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:00.493208 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.493191    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-781009b5-a2c7-4181-b106-aa18d6760848" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc") on node "ip-10-0-132-0.ec2.internal"
Sep 17 18:52:00.499478 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.499450    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f640c92c-8e6c-4fa3-8147-2cf1951e370a" (UID: "f640c92c-8e6c-4fa3-8147-2cf1951e370a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:00.578501 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.578470    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:52:00.578501 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.578495    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:52:00.578501 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.578506    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-781009b5-a2c7-4181-b106-aa18d6760848\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ed4cf9c88c8692bc\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"csi-ed995ece8c6dd2a8e1c7c5c0a4926b43ee80d8d41833eeb850296e8928cec2c8\""
Sep 17 18:52:00.578501 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.578528    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2nrs\" (UniqueName: \"kubernetes.io/projected/f640c92c-8e6c-4fa3-8147-2cf1951e370a-kube-api-access-x2nrs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:52:00.578501 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.578537    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f640c92c-8e6c-4fa3-8147-2cf1951e370a-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:52:00.744189 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.744156    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"]
Sep 17 18:52:00.747580 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:00.747548    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7dc94b9d9d-gcf42"]
Sep 17 18:52:01.938862 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:01.938829    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" path="/var/lib/kubelet/pods/f640c92c-8e6c-4fa3-8147-2cf1951e370a/volumes"
Sep 17 18:52:04.797469 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:04.797437    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:04.897164 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:04.897103    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:04.997976 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:04.997943    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:05.168937 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:05.168861    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:05.290145 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:05.290112    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:05.921466 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:05.921420    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 18:52:07.339028 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:07.338994    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:07.408812 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:07.408778    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:08.081762 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.081723    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"]
Sep 17 18:52:08.082164 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.082144    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerName="main"
Sep 17 18:52:08.082265 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.082166    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerName="main"
Sep 17 18:52:08.082265 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.082180    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerName="storage-initializer"
Sep 17 18:52:08.082265 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.082190    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerName="storage-initializer"
Sep 17 18:52:08.082421 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.082290    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f640c92c-8e6c-4fa3-8147-2cf1951e370a" containerName="main"
Sep 17 18:52:08.090594 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.088021    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"
Sep 17 18:52:08.091723 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.091690    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 17 18:52:08.092671 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.092601    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 18:52:08.092798 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.092679    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 18:52:08.092798 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.092739    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-tkxrd\""
Sep 17 18:52:08.094931 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.094901    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"]
Sep 17 18:52:08.181191 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.181160    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:08.241831 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.241801    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rphvd\" (UniqueName: \"kubernetes.io/projected/b4ae6852-9ad9-48d0-b83d-67747f1cbc1e-kube-api-access-rphvd\") pod \"e2e-pvc-lora-download-tm77s\" (UID: \"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"
Sep 17 18:52:08.342906 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.342827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rphvd\" (UniqueName: \"kubernetes.io/projected/b4ae6852-9ad9-48d0-b83d-67747f1cbc1e-kube-api-access-rphvd\") pod \"e2e-pvc-lora-download-tm77s\" (UID: \"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"
Sep 17 18:52:08.351587 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:08.351558    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rphvd\" (UniqueName: \"kubernetes.io/projected/b4ae6852-9ad9-48d0-b83d-67747f1cbc1e-kube-api-access-rphvd\") pod \"e2e-pvc-lora-download-tm77s\" (UID: \"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"
Sep 17 18:52:09.769064 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:09.769032    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"]
Sep 17 18:52:09.772794 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:09.772768    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-5mzgz"]
Sep 17 18:52:09.939558 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:09.939459    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6b30beb-fe9d-4bfc-9e43-081def206fa9" path="/var/lib/kubelet/pods/f6b30beb-fe9d-4bfc-9e43-081def206fa9/volumes"
Sep 17 18:52:10.561892 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:10.561852    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"e2e-pvc-lora-download-tm77s\" (UID: \"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"
Sep 17 18:52:10.567267 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:10.567235    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"e2e-pvc-lora-download-tm77s\" (UID: \"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"
Sep 17 18:52:10.644718 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:10.644684    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:10.663271 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:10.663234    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"e2e-pvc-lora-download-tm77s\" (UID: \"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"
Sep 17 18:52:10.663413 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:10.663369    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"e2e-pvc-lora-download-tm77s\" (UID: \"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"
Sep 17 18:52:10.665937 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:10.665916    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"e2e-pvc-lora-download-tm77s\" (UID: \"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e\") DevicePath \"csi-6367fbbbc1c99ff3b9067ef0dab92b08b6b8562a7ac6a24c6e68231cbfd2b924\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"
Sep 17 18:52:10.708728 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:10.708689    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:10.875614 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:10.875572    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"e2e-pvc-lora-download-tm77s\" (UID: \"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/383685ecf59044660dae27c917925f3a8a559dbbf825118fc113143f6ed1cdfa/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"
Sep 17 18:52:10.890663 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:10.890625    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"e2e-pvc-lora-download-tm77s\" (UID: \"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"
Sep 17 18:52:11.100000 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:11.099957    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"
Sep 17 18:52:11.162165 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:11.162096    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:11.377988 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:11.377956    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"]
Sep 17 18:52:11.380024 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:52:11.379995    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4ae6852_9ad9_48d0_b83d_67747f1cbc1e.slice/crio-0e0a3f278c2e977817b57517bfbdeeddb7c6eec1e459e1a658805ed9fc1a6785 WatchSource:0}: Error finding container 0e0a3f278c2e977817b57517bfbdeeddb7c6eec1e459e1a658805ed9fc1a6785: Status 404 returned error can't find the container with id 0e0a3f278c2e977817b57517bfbdeeddb7c6eec1e459e1a658805ed9fc1a6785
Sep 17 18:52:11.485896 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:11.485854    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s" event={"ID":"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e","Type":"ContainerStarted","Data":"acbd5c175d48535bc81d067bc47324e1dda123791dd80f0b1ebae396b872da4a"}
Sep 17 18:52:11.486034 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:11.485908    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s" event={"ID":"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e","Type":"ContainerStarted","Data":"0e0a3f278c2e977817b57517bfbdeeddb7c6eec1e459e1a658805ed9fc1a6785"}
Sep 17 18:52:11.500155 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:11.500111    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s" podStartSLOduration=6.500096147 podStartE2EDuration="6.500096147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:11.499878028 +0000 UTC m=+3301.800700353" watchObservedRunningTime="2026-09-17 18:52:11.500096147 +0000 UTC m=+3301.800918470"
Sep 17 18:52:12.139628 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:12.139594    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:52:12.156074 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:12.156047    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:52:13.500912 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:13.500883    2574 generic.go:356] "Generic (PLEG): container finished" podID="b4ae6852-9ad9-48d0-b83d-67747f1cbc1e" containerID="acbd5c175d48535bc81d067bc47324e1dda123791dd80f0b1ebae396b872da4a" exitCode=0
Sep 17 18:52:13.501246 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:13.500962    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s" event={"ID":"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e","Type":"ContainerDied","Data":"acbd5c175d48535bc81d067bc47324e1dda123791dd80f0b1ebae396b872da4a"}
Sep 17 18:52:13.803097 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:13.803063    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:13.904332 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:13.904298    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:14.645461 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:14.645437    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"
Sep 17 18:52:14.804899 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:14.804818    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e\" (UID: \"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e\") "
Sep 17 18:52:14.804899 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:14.804886    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rphvd\" (UniqueName: \"kubernetes.io/projected/b4ae6852-9ad9-48d0-b83d-67747f1cbc1e-kube-api-access-rphvd\") pod \"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e\" (UID: \"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e\") "
Sep 17 18:52:14.807551 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:14.807493    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ae6852-9ad9-48d0-b83d-67747f1cbc1e-kube-api-access-rphvd" (OuterVolumeSpecName: "kube-api-access-rphvd") pod "b4ae6852-9ad9-48d0-b83d-67747f1cbc1e" (UID: "b4ae6852-9ad9-48d0-b83d-67747f1cbc1e"). InnerVolumeSpecName "kube-api-access-rphvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:52:14.807771 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:14.807671    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c" (OuterVolumeSpecName: "model-storage") pod "b4ae6852-9ad9-48d0-b83d-67747f1cbc1e" (UID: "b4ae6852-9ad9-48d0-b83d-67747f1cbc1e"). InnerVolumeSpecName "pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:52:14.905859 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:14.905812    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") on node \"ip-10-0-132-0.ec2.internal\" "
Sep 17 18:52:14.905859 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:14.905857    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rphvd\" (UniqueName: \"kubernetes.io/projected/b4ae6852-9ad9-48d0-b83d-67747f1cbc1e-kube-api-access-rphvd\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:52:15.513162 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:15.513128    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"
Sep 17 18:52:15.513162 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:15.513139    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s" event={"ID":"b4ae6852-9ad9-48d0-b83d-67747f1cbc1e","Type":"ContainerDied","Data":"0e0a3f278c2e977817b57517bfbdeeddb7c6eec1e459e1a658805ed9fc1a6785"}
Sep 17 18:52:15.513342 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:15.513177    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e0a3f278c2e977817b57517bfbdeeddb7c6eec1e459e1a658805ed9fc1a6785"
Sep 17 18:52:15.876239 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:15.876213    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c") on node "ip-10-0-132-0.ec2.internal"
Sep 17 18:52:15.915497 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:15.915460    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"csi-6367fbbbc1c99ff3b9067ef0dab92b08b6b8562a7ac6a24c6e68231cbfd2b924\""
Sep 17 18:52:15.920646 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:15.920618    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 18:52:17.539240 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:17.539202    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:17.639264 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:17.639223    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:25.921018 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:25.920955    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 18:52:32.659401 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.659359    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"]
Sep 17 18:52:32.660048 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.659784    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ae6852-9ad9-48d0-b83d-67747f1cbc1e" containerName="storage-initializer"
Sep 17 18:52:32.660048 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.659799    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ae6852-9ad9-48d0-b83d-67747f1cbc1e" containerName="storage-initializer"
Sep 17 18:52:32.660048 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.659861    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ae6852-9ad9-48d0-b83d-67747f1cbc1e" containerName="storage-initializer"
Sep 17 18:52:32.665691 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.665662    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.668381 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.668204    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 18:52:32.668381 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.668213    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-tkxrd\""
Sep 17 18:52:32.668381 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.668247    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 18:52:32.668381 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.668280    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 17 18:52:32.674641 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.674613    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"]
Sep 17 18:52:32.757870 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.757839    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.758040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.757897    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be88994-19f7-4515-bb8d-f902125adacf-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.758040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.757929    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.758040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.757972    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.758040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.758034    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.859033 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.858997    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be88994-19f7-4515-bb8d-f902125adacf-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.859033 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.859032    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.859256 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.859058    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.859256 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.859109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.859256 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.859173    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.859256 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.859206    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fslk5\" (UniqueName: \"kubernetes.io/projected/9be88994-19f7-4515-bb8d-f902125adacf-kube-api-access-fslk5\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.859503 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.859481    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.859503 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.859493    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.859646 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.859610    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.861719 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.861695    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.861959 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.861937    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be88994-19f7-4515-bb8d-f902125adacf-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.959987 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.959913    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fslk5\" (UniqueName: \"kubernetes.io/projected/9be88994-19f7-4515-bb8d-f902125adacf-kube-api-access-fslk5\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:32.968695 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:32.968673    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fslk5\" (UniqueName: \"kubernetes.io/projected/9be88994-19f7-4515-bb8d-f902125adacf-kube-api-access-fslk5\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:33.354682 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:33.354642    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:35.921227 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:35.921179    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 18:52:40.720713 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:40.720676    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:40.726872 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:40.726822    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:40.820978 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:40.820949    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:40.821102 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:40.821076    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:40.827486 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:40.827462    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") DevicePath \"csi-6367fbbbc1c99ff3b9067ef0dab92b08b6b8562a7ac6a24c6e68231cbfd2b924\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:40.916147 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:40.916122    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/383685ecf59044660dae27c917925f3a8a559dbbf825118fc113143f6ed1cdfa/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:40.935768 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:40.935744    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:41.079694 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:41.079666    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:41.359610 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:41.359581    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"]
Sep 17 18:52:41.361229 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:52:41.361200    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be88994_19f7_4515_bb8d_f902125adacf.slice/crio-5952d749b84915cd46994ee3057baa5305557919aa36d1b7c242b605071c56df WatchSource:0}: Error finding container 5952d749b84915cd46994ee3057baa5305557919aa36d1b7c242b605071c56df: Status 404 returned error can't find the container with id 5952d749b84915cd46994ee3057baa5305557919aa36d1b7c242b605071c56df
Sep 17 18:52:41.678898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:41.678813    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb" event={"ID":"9be88994-19f7-4515-bb8d-f902125adacf","Type":"ContainerStarted","Data":"004d0fb6be426a422d44ef5d9dd6082be3e71c97ae9529dc753f9e91e51555a5"}
Sep 17 18:52:41.678898 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:41.678889    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb" event={"ID":"9be88994-19f7-4515-bb8d-f902125adacf","Type":"ContainerStarted","Data":"5952d749b84915cd46994ee3057baa5305557919aa36d1b7c242b605071c56df"}
Sep 17 18:52:41.703931 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:41.703879    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb" podStartSLOduration=9.703860715 podStartE2EDuration="9.703860715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:41.700901882 +0000 UTC m=+3332.001724203" watchObservedRunningTime="2026-09-17 18:52:41.703860715 +0000 UTC m=+3332.004683041"
Sep 17 18:52:42.079786 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:42.079753    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:45.938334 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:45.938302    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:52:45.949235 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:45.949201    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:52:51.080703 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:51.080652    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:52:51.082443 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:52:51.082403    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb" podUID="9be88994-19f7-4515-bb8d-f902125adacf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 18:53:01.080372 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:01.080323    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb" podUID="9be88994-19f7-4515-bb8d-f902125adacf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 18:53:07.260392 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:07.260347    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"]
Sep 17 18:53:07.265752 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:07.265684    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:07.373542 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:07.373491    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:08.692445 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:08.692412    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:08.793358 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:08.793321    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:11.080177 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:11.080131    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb" podUID="9be88994-19f7-4515-bb8d-f902125adacf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 18:53:21.080373 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:21.080332    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb" podUID="9be88994-19f7-4515-bb8d-f902125adacf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 18:53:22.285849 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:22.285809    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main" containerID="cri-o://ff3b26037a00d530a11c7bd6cac2735bc8419a4e44cc417f62ad490f24a25993" gracePeriod=45
Sep 17 18:53:22.915849 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:22.915806    2574 generic.go:356] "Generic (PLEG): container finished" podID="616d1c70-058d-4d94-8106-ae939ecba97d" containerID="ff3b26037a00d530a11c7bd6cac2735bc8419a4e44cc417f62ad490f24a25993" exitCode=0
Sep 17 18:53:22.916007 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:22.915865    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" event={"ID":"616d1c70-058d-4d94-8106-ae939ecba97d","Type":"ContainerDied","Data":"ff3b26037a00d530a11c7bd6cac2735bc8419a4e44cc417f62ad490f24a25993"}
Sep 17 18:53:23.044023 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.043992    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:53:23.192182 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.192091    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/616d1c70-058d-4d94-8106-ae939ecba97d-cabundle-cert\") pod \"616d1c70-058d-4d94-8106-ae939ecba97d\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") "
Sep 17 18:53:23.192182 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.192148    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s5qx\" (UniqueName: \"kubernetes.io/projected/616d1c70-058d-4d94-8106-ae939ecba97d-kube-api-access-6s5qx\") pod \"616d1c70-058d-4d94-8106-ae939ecba97d\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") "
Sep 17 18:53:23.192426 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.192188    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-kserve-provision-location\") pod \"616d1c70-058d-4d94-8106-ae939ecba97d\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") "
Sep 17 18:53:23.192426 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.192218    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-dshm\") pod \"616d1c70-058d-4d94-8106-ae939ecba97d\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") "
Sep 17 18:53:23.192426 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.192334    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616d1c70-058d-4d94-8106-ae939ecba97d-tls-certs\") pod \"616d1c70-058d-4d94-8106-ae939ecba97d\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") "
Sep 17 18:53:23.192426 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.192376    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-tmp-dir\") pod \"616d1c70-058d-4d94-8106-ae939ecba97d\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") "
Sep 17 18:53:23.192426 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.192417    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-home\") pod \"616d1c70-058d-4d94-8106-ae939ecba97d\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") "
Sep 17 18:53:23.192735 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.192450    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-model-cache\") pod \"616d1c70-058d-4d94-8106-ae939ecba97d\" (UID: \"616d1c70-058d-4d94-8106-ae939ecba97d\") "
Sep 17 18:53:23.192735 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.192593    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616d1c70-058d-4d94-8106-ae939ecba97d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "616d1c70-058d-4d94-8106-ae939ecba97d" (UID: "616d1c70-058d-4d94-8106-ae939ecba97d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:53:23.192967 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.192898    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-model-cache" (OuterVolumeSpecName: "model-cache") pod "616d1c70-058d-4d94-8106-ae939ecba97d" (UID: "616d1c70-058d-4d94-8106-ae939ecba97d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:23.193229 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.193212    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.193307 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.193236    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/616d1c70-058d-4d94-8106-ae939ecba97d-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.193492 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.193465    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-home" (OuterVolumeSpecName: "home") pod "616d1c70-058d-4d94-8106-ae939ecba97d" (UID: "616d1c70-058d-4d94-8106-ae939ecba97d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:23.195294 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.195270    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-dshm" (OuterVolumeSpecName: "dshm") pod "616d1c70-058d-4d94-8106-ae939ecba97d" (UID: "616d1c70-058d-4d94-8106-ae939ecba97d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:23.195294 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.195290    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616d1c70-058d-4d94-8106-ae939ecba97d-kube-api-access-6s5qx" (OuterVolumeSpecName: "kube-api-access-6s5qx") pod "616d1c70-058d-4d94-8106-ae939ecba97d" (UID: "616d1c70-058d-4d94-8106-ae939ecba97d"). InnerVolumeSpecName "kube-api-access-6s5qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:53:23.195919 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.195904    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/616d1c70-058d-4d94-8106-ae939ecba97d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "616d1c70-058d-4d94-8106-ae939ecba97d" (UID: "616d1c70-058d-4d94-8106-ae939ecba97d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:53:23.214319 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.214248    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "616d1c70-058d-4d94-8106-ae939ecba97d" (UID: "616d1c70-058d-4d94-8106-ae939ecba97d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:23.215776 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.215747    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "616d1c70-058d-4d94-8106-ae939ecba97d" (UID: "616d1c70-058d-4d94-8106-ae939ecba97d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:23.293844 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.293806    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.293844 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.293837    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.293844 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.293848    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s5qx\" (UniqueName: \"kubernetes.io/projected/616d1c70-058d-4d94-8106-ae939ecba97d-kube-api-access-6s5qx\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.294290 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.293858    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.294290 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.293866    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616d1c70-058d-4d94-8106-ae939ecba97d-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.294290 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.293874    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616d1c70-058d-4d94-8106-ae939ecba97d-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.923549 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.923502    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"
Sep 17 18:53:23.923549 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.923535    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7" event={"ID":"616d1c70-058d-4d94-8106-ae939ecba97d","Type":"ContainerDied","Data":"9e6e5e3ef9dd57cbe7c35d39deed97f90b7b8b2084cc19d14d45a493604c5f7e"}
Sep 17 18:53:23.923771 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.923625    2574 scope.go:117] "RemoveContainer" containerID="ff3b26037a00d530a11c7bd6cac2735bc8419a4e44cc417f62ad490f24a25993"
Sep 17 18:53:23.935266 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.935246    2574 scope.go:117] "RemoveContainer" containerID="09c3ae528060d14ec788b17e88c2ea4144eb25b80756b52941a2e60cedd983ee"
Sep 17 18:53:23.946723 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.946698    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"]
Sep 17 18:53:23.951880 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:23.951858    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7f4fc6ddd9-sx7r7"]
Sep 17 18:53:25.410367 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:25.410333    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:25.938101 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:25.938066    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" path="/var/lib/kubelet/pods/616d1c70-058d-4d94-8106-ae939ecba97d/volumes"
Sep 17 18:53:31.080150 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:31.080101    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb" podUID="9be88994-19f7-4515-bb8d-f902125adacf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 18:53:41.080292 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:41.080249    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb" podUID="9be88994-19f7-4515-bb8d-f902125adacf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 18:53:51.080688 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:51.080607    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb" podUID="9be88994-19f7-4515-bb8d-f902125adacf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 18:53:55.932962 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:55.932922    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:56.032572 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:53:56.032507    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:00.607860 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:00.607825    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:00.940196 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:00.940107    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:01.090199 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:01.090169    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:54:01.097686 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:01.097646    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:54:02.118499 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:02.118466    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:23.134990 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:23.134952    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:23.555374 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:23.555330    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:29.560835 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:29.560796    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:33.340484 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:33.340443    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb_9be88994-19f7-4515-bb8d-f902125adacf/main/0.log"
Sep 17 18:54:34.359156 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:34.359122    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"]
Sep 17 18:54:34.382103 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:34.382077    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:38.876187 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:38.876151    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:38.976374 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:38.976341    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:44.557831 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:44.557793    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:44.889307 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:44.889215    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:44.970189 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:44.970136    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:49.382613 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:49.382564    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb" podUID="9be88994-19f7-4515-bb8d-f902125adacf" containerName="main" containerID="cri-o://004d0fb6be426a422d44ef5d9dd6082be3e71c97ae9529dc753f9e91e51555a5" gracePeriod=45
Sep 17 18:54:50.048373 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.048345    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:54:50.178061 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.177975    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-model-cache\") pod \"9be88994-19f7-4515-bb8d-f902125adacf\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") "
Sep 17 18:54:50.178061 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.178027    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-home\") pod \"9be88994-19f7-4515-bb8d-f902125adacf\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") "
Sep 17 18:54:50.178061 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.178051    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-dshm\") pod \"9be88994-19f7-4515-bb8d-f902125adacf\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") "
Sep 17 18:54:50.178301 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.178155    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") pod \"9be88994-19f7-4515-bb8d-f902125adacf\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") "
Sep 17 18:54:50.178301 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.178183    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-tmp-dir\") pod \"9be88994-19f7-4515-bb8d-f902125adacf\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") "
Sep 17 18:54:50.178301 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.178243    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be88994-19f7-4515-bb8d-f902125adacf-tls-certs\") pod \"9be88994-19f7-4515-bb8d-f902125adacf\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") "
Sep 17 18:54:50.178301 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.178291    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-model-cache" (OuterVolumeSpecName: "model-cache") pod "9be88994-19f7-4515-bb8d-f902125adacf" (UID: "9be88994-19f7-4515-bb8d-f902125adacf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:50.178486 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.178355    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fslk5\" (UniqueName: \"kubernetes.io/projected/9be88994-19f7-4515-bb8d-f902125adacf-kube-api-access-fslk5\") pod \"9be88994-19f7-4515-bb8d-f902125adacf\" (UID: \"9be88994-19f7-4515-bb8d-f902125adacf\") "
Sep 17 18:54:50.178486 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.178452    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-home" (OuterVolumeSpecName: "home") pod "9be88994-19f7-4515-bb8d-f902125adacf" (UID: "9be88994-19f7-4515-bb8d-f902125adacf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:50.178672 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.178646    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:54:50.178672 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.178670    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:54:50.180440 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.180414    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-dshm" (OuterVolumeSpecName: "dshm") pod "9be88994-19f7-4515-bb8d-f902125adacf" (UID: "9be88994-19f7-4515-bb8d-f902125adacf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:50.180588 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.180563    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be88994-19f7-4515-bb8d-f902125adacf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9be88994-19f7-4515-bb8d-f902125adacf" (UID: "9be88994-19f7-4515-bb8d-f902125adacf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:50.180651 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.180613    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be88994-19f7-4515-bb8d-f902125adacf-kube-api-access-fslk5" (OuterVolumeSpecName: "kube-api-access-fslk5") pod "9be88994-19f7-4515-bb8d-f902125adacf" (UID: "9be88994-19f7-4515-bb8d-f902125adacf"). InnerVolumeSpecName "kube-api-access-fslk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:50.186779 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.186758    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c" (OuterVolumeSpecName: "kserve-pvc-source") pod "9be88994-19f7-4515-bb8d-f902125adacf" (UID: "9be88994-19f7-4515-bb8d-f902125adacf"). InnerVolumeSpecName "pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:54:50.190772 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.190742    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9be88994-19f7-4515-bb8d-f902125adacf" (UID: "9be88994-19f7-4515-bb8d-f902125adacf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:50.279318 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.279291    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be88994-19f7-4515-bb8d-f902125adacf-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:54:50.279400 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.279320    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fslk5\" (UniqueName: \"kubernetes.io/projected/9be88994-19f7-4515-bb8d-f902125adacf-kube-api-access-fslk5\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:54:50.279400 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.279331    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:54:50.279400 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.279356    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") on node \"ip-10-0-132-0.ec2.internal\" "
Sep 17 18:54:50.279400 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.279366    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be88994-19f7-4515-bb8d-f902125adacf-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:54:50.308111 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.308096    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c") on node "ip-10-0-132-0.ec2.internal"
Sep 17 18:54:50.380229 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.380202    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-de73aedd-ab9c-4163-ada6-10615b7d9c8b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d6627a97ce29246c\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"csi-6367fbbbc1c99ff3b9067ef0dab92b08b6b8562a7ac6a24c6e68231cbfd2b924\""
Sep 17 18:54:50.408053 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.408025    2574 generic.go:356] "Generic (PLEG): container finished" podID="9be88994-19f7-4515-bb8d-f902125adacf" containerID="004d0fb6be426a422d44ef5d9dd6082be3e71c97ae9529dc753f9e91e51555a5" exitCode=0
Sep 17 18:54:50.408350 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.408080    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"
Sep 17 18:54:50.408350 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.408111    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb" event={"ID":"9be88994-19f7-4515-bb8d-f902125adacf","Type":"ContainerDied","Data":"004d0fb6be426a422d44ef5d9dd6082be3e71c97ae9529dc753f9e91e51555a5"}
Sep 17 18:54:50.408350 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.408165    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb" event={"ID":"9be88994-19f7-4515-bb8d-f902125adacf","Type":"ContainerDied","Data":"5952d749b84915cd46994ee3057baa5305557919aa36d1b7c242b605071c56df"}
Sep 17 18:54:50.408350 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.408197    2574 scope.go:117] "RemoveContainer" containerID="004d0fb6be426a422d44ef5d9dd6082be3e71c97ae9529dc753f9e91e51555a5"
Sep 17 18:54:50.418278 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.418251    2574 scope.go:117] "RemoveContainer" containerID="004d0fb6be426a422d44ef5d9dd6082be3e71c97ae9529dc753f9e91e51555a5"
Sep 17 18:54:50.418550 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:54:50.418503    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004d0fb6be426a422d44ef5d9dd6082be3e71c97ae9529dc753f9e91e51555a5\": container with ID starting with 004d0fb6be426a422d44ef5d9dd6082be3e71c97ae9529dc753f9e91e51555a5 not found: ID does not exist" containerID="004d0fb6be426a422d44ef5d9dd6082be3e71c97ae9529dc753f9e91e51555a5"
Sep 17 18:54:50.418630 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.418555    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004d0fb6be426a422d44ef5d9dd6082be3e71c97ae9529dc753f9e91e51555a5"} err="failed to get container status \"004d0fb6be426a422d44ef5d9dd6082be3e71c97ae9529dc753f9e91e51555a5\": rpc error: code = NotFound desc = could not find container \"004d0fb6be426a422d44ef5d9dd6082be3e71c97ae9529dc753f9e91e51555a5\": container with ID starting with 004d0fb6be426a422d44ef5d9dd6082be3e71c97ae9529dc753f9e91e51555a5 not found: ID does not exist"
Sep 17 18:54:50.430692 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.430645    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"]
Sep 17 18:54:50.434004 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:50.433981    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sl9bb"]
Sep 17 18:54:51.938471 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:51.938440    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9be88994-19f7-4515-bb8d-f902125adacf" path="/var/lib/kubelet/pods/9be88994-19f7-4515-bb8d-f902125adacf/volumes"
Sep 17 18:54:54.513425 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:54.513396    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:54.614364 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:54.614331    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:59.491678 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:59.491635    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"]
Sep 17 18:54:59.496563 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:59.496534    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-tm77s"]
Sep 17 18:54:59.937978 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:54:59.937939    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4ae6852-9ad9-48d0-b83d-67747f1cbc1e" path="/var/lib/kubelet/pods/b4ae6852-9ad9-48d0-b83d-67747f1cbc1e/volumes"
Sep 17 18:55:00.514094 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:00.514051    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:00.576020 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:00.575984    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:00.680325 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:00.680290    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:00.782057 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:00.781961    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:06.690571 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:06.690535    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:06.791717 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:06.791677    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:12.980978 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.980941    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"]
Sep 17 18:55:12.981345 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.981289    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main"
Sep 17 18:55:12.981345 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.981299    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main"
Sep 17 18:55:12.981345 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.981307    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="storage-initializer"
Sep 17 18:55:12.981345 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.981313    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="storage-initializer"
Sep 17 18:55:12.981345 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.981320    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9be88994-19f7-4515-bb8d-f902125adacf" containerName="main"
Sep 17 18:55:12.981345 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.981326    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9be88994-19f7-4515-bb8d-f902125adacf" containerName="main"
Sep 17 18:55:12.981560 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.981377    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="616d1c70-058d-4d94-8106-ae939ecba97d" containerName="main"
Sep 17 18:55:12.981560 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.981387    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9be88994-19f7-4515-bb8d-f902125adacf" containerName="main"
Sep 17 18:55:12.983446 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.983429    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:12.986234 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.986206    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 18:55:12.986397 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.986213    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:55:12.986397 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.986282    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 18:55:12.986397 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.986325    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-x4g28\""
Sep 17 18:55:12.986598 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.986411    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 18:55:12.986960 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.986940    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 17 18:55:12.998553 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:12.996715    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"]
Sep 17 18:55:13.004357 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.004328    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:13.055760 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.055722    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2knr9\" (UniqueName: \"kubernetes.io/projected/3f343194-783d-45c7-9440-f9cc54889cbb-kube-api-access-2knr9\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.055936 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.055768    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-model-cache\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.055936 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.055828    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-kserve-provision-location\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.055936 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.055847    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-tmp-dir\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.055936 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.055912    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-home\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.056154 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.056065    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-dshm\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.056154 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.056092    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f343194-783d-45c7-9440-f9cc54889cbb-tls-certs\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.056154 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.056129    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f343194-783d-45c7-9440-f9cc54889cbb-cabundle-cert\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.157465 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.157436    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-kserve-provision-location\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.157621 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.157479    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-tmp-dir\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.157621 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.157508    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-home\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.157621 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.157570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-dshm\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.157621 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.157593    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f343194-783d-45c7-9440-f9cc54889cbb-tls-certs\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.157852 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.157624    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f343194-783d-45c7-9440-f9cc54889cbb-cabundle-cert\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.157852 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.157663    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2knr9\" (UniqueName: \"kubernetes.io/projected/3f343194-783d-45c7-9440-f9cc54889cbb-kube-api-access-2knr9\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.157852 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.157690    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-model-cache\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.158012 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.157904    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-tmp-dir\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.158012 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.157986    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-home\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.158139 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.158119    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-model-cache\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.158186 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.158169    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-kserve-provision-location\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.158486 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.158456    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f343194-783d-45c7-9440-f9cc54889cbb-cabundle-cert\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.160155 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.160128    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-dshm\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.160340 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.160318    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f343194-783d-45c7-9440-f9cc54889cbb-tls-certs\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.165967 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.165946    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2knr9\" (UniqueName: \"kubernetes.io/projected/3f343194-783d-45c7-9440-f9cc54889cbb-kube-api-access-2knr9\") pod \"tls-verification-test-kserve-6594d79dc8-hzwnj\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.294866 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.294832    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:13.301226 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.301203    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:13.574456 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:13.574420    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"]
Sep 17 18:55:13.576148 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:55:13.576119    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f343194_783d_45c7_9440_f9cc54889cbb.slice/crio-d778d3c5c18c674be0cf75e3430970fe6bef849fae875f666b47e1d454c73cb9 WatchSource:0}: Error finding container d778d3c5c18c674be0cf75e3430970fe6bef849fae875f666b47e1d454c73cb9: Status 404 returned error can't find the container with id d778d3c5c18c674be0cf75e3430970fe6bef849fae875f666b47e1d454c73cb9
Sep 17 18:55:14.538680 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:14.538647    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" event={"ID":"3f343194-783d-45c7-9440-f9cc54889cbb","Type":"ContainerStarted","Data":"bb7c9b78205487bca82e648d12926b128bd578bc5bdc6685859945395cc3e366"}
Sep 17 18:55:14.539041 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:14.538698    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" event={"ID":"3f343194-783d-45c7-9440-f9cc54889cbb","Type":"ContainerStarted","Data":"d778d3c5c18c674be0cf75e3430970fe6bef849fae875f666b47e1d454c73cb9"}
Sep 17 18:55:16.552465 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:16.552427    2574 generic.go:356] "Generic (PLEG): container finished" podID="3f343194-783d-45c7-9440-f9cc54889cbb" containerID="bb7c9b78205487bca82e648d12926b128bd578bc5bdc6685859945395cc3e366" exitCode=0
Sep 17 18:55:16.552850 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:16.552500    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" event={"ID":"3f343194-783d-45c7-9440-f9cc54889cbb","Type":"ContainerDied","Data":"bb7c9b78205487bca82e648d12926b128bd578bc5bdc6685859945395cc3e366"}
Sep 17 18:55:17.560805 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:17.560770    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" event={"ID":"3f343194-783d-45c7-9440-f9cc54889cbb","Type":"ContainerStarted","Data":"623c134936f3d57d6646b5be41ad7b9647b7f1c5e4d873ac222c1c3448f39cac"}
Sep 17 18:55:17.587343 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:17.587282    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" podStartSLOduration=3.442510476 podStartE2EDuration="5.58726213s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.144751654s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:17.584811539 +0000 UTC m=+3487.885633864" watchObservedRunningTime="2026-09-17 18:55:17.58726213 +0000 UTC m=+3487.888084455"
Sep 17 18:55:18.295701 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:18.295667    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:23.295995 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:23.295957    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:55:23.298001 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:23.297969    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 17 18:55:30.779530 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:30.779486    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:33.296082 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:33.296032    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 17 18:55:42.890183 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:42.890150    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:43.295822 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:43.295787    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 17 18:55:53.296191 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:55:53.296137    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 17 18:56:03.296212 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:03.296162    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 17 18:56:05.345274 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:05.345240    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:13.296239 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:13.296190    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 17 18:56:21.349096 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:21.349056    2574 scope.go:117] "RemoveContainer" containerID="c4fa267475e89290e9982a3c2ef4ff7e57ae88f10598c7ae9d801bcdd3bcbeb2"
Sep 17 18:56:23.296311 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:23.296263    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 17 18:56:25.394605 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:25.394570    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:25.577058 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:25.577026    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:31.997328 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:31.997293    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:33.295509 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:33.295471    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 17 18:56:35.114691 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.114647    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:35.263676 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.263638    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:35.382105 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.382022    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:35.466129 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.466099    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:56:35.471303 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.471280    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.473971 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.473940    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 18:56:35.474217 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.474194    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 18:56:35.474407 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.474219    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:56:35.474529 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.474257    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 18:56:35.474975 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.474955    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-fhnpt\""
Sep 17 18:56:35.475073 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.474979    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-ld4jx\""
Sep 17 18:56:35.475073 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.474992    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 18:56:35.481243 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.481221    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:56:35.538753 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.538722    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:35.643683 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.643591    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.643683 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.643627    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d306c934-15fd-4b88-b641-ffe2aa31049b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.643683 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.643654    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.643919 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.643713    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.643919 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.643733    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr5kn\" (UniqueName: \"kubernetes.io/projected/d306c934-15fd-4b88-b641-ffe2aa31049b-kube-api-access-fr5kn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.643919 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.643760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.643919 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.643807    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d306c934-15fd-4b88-b641-ffe2aa31049b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.643919 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.643837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.744850 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.744811    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.745040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.744860    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.745040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.744893    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr5kn\" (UniqueName: \"kubernetes.io/projected/d306c934-15fd-4b88-b641-ffe2aa31049b-kube-api-access-fr5kn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.745040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.744935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.745040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.744985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d306c934-15fd-4b88-b641-ffe2aa31049b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.745040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.745021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.745320 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.745057    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.745320 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.745219    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.745320 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.745263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d306c934-15fd-4b88-b641-ffe2aa31049b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.745320 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.745299    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.746116 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.746071    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.746254 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.746127    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.746582 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.746557    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d306c934-15fd-4b88-b641-ffe2aa31049b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.748721 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.748695    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.753350 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.753325    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr5kn\" (UniqueName: \"kubernetes.io/projected/d306c934-15fd-4b88-b641-ffe2aa31049b-kube-api-access-fr5kn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.755925 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.755899    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d306c934-15fd-4b88-b641-ffe2aa31049b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:35.783628 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:35.783605    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:56:36.060464 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:36.060435    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:56:36.062764 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:56:36.062733    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd306c934_15fd_4b88_b641_ffe2aa31049b.slice/crio-bf12adba5294bf23bdf988f8a2b05c96abf356f5b2c64be21062cb78df530d13 WatchSource:0}: Error finding container bf12adba5294bf23bdf988f8a2b05c96abf356f5b2c64be21062cb78df530d13: Status 404 returned error can't find the container with id bf12adba5294bf23bdf988f8a2b05c96abf356f5b2c64be21062cb78df530d13
Sep 17 18:56:37.006291 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:37.006234    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d306c934-15fd-4b88-b641-ffe2aa31049b","Type":"ContainerStarted","Data":"7c7dbade9a968c6bdcabf53037ae55c726a270ceb847755b7d21ac586c0bdf7d"}
Sep 17 18:56:37.006935 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:37.006322    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d306c934-15fd-4b88-b641-ffe2aa31049b","Type":"ContainerStarted","Data":"bf12adba5294bf23bdf988f8a2b05c96abf356f5b2c64be21062cb78df530d13"}
Sep 17 18:56:38.366916 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:38.366880    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:38.466828 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:38.466795    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:39.020154 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:39.020121    2574 generic.go:356] "Generic (PLEG): container finished" podID="d306c934-15fd-4b88-b641-ffe2aa31049b" containerID="7c7dbade9a968c6bdcabf53037ae55c726a270ceb847755b7d21ac586c0bdf7d" exitCode=0
Sep 17 18:56:39.020385 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:39.020201    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d306c934-15fd-4b88-b641-ffe2aa31049b","Type":"ContainerDied","Data":"7c7dbade9a968c6bdcabf53037ae55c726a270ceb847755b7d21ac586c0bdf7d"}
Sep 17 18:56:40.027748 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:40.027712    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d306c934-15fd-4b88-b641-ffe2aa31049b","Type":"ContainerStarted","Data":"e6b41e7cf7597593cdc8835f84ed06c8697f63f133bda1024f957722ce979b5f"}
Sep 17 18:56:40.051737 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:40.051685    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.769599769 podStartE2EDuration="5.051668651s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.282068882s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:40.049297776 +0000 UTC m=+3570.350120110" watchObservedRunningTime="2026-09-17 18:56:40.051668651 +0000 UTC m=+3570.352490975"
Sep 17 18:56:43.295385 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:43.295334    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 17 18:56:45.361399 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:45.361361    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:53.296028 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:56:53.295943    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 17 18:57:03.295979 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:03.295933    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 17 18:57:12.181676 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:12.181646    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:57:12.198284 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:12.198248    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 18:57:13.305452 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:13.305413    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:57:13.313996 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:13.313965    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:57:18.098217 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:18.098175    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:18.112543 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:18.112483    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"]
Sep 17 18:57:19.509263 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:19.509225    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:33.138134 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:33.138086    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main" containerID="cri-o://623c134936f3d57d6646b5be41ad7b9647b7f1c5e4d873ac222c1c3448f39cac" gracePeriod=45
Sep 17 18:57:33.306892 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:33.306846    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 17 18:57:34.121553 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.121504    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:57:34.154486 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.154454    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-model-cache\") pod \"3f343194-783d-45c7-9440-f9cc54889cbb\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") "
Sep 17 18:57:34.154908 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.154540    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f343194-783d-45c7-9440-f9cc54889cbb-cabundle-cert\") pod \"3f343194-783d-45c7-9440-f9cc54889cbb\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") "
Sep 17 18:57:34.154908 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.154577    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f343194-783d-45c7-9440-f9cc54889cbb-tls-certs\") pod \"3f343194-783d-45c7-9440-f9cc54889cbb\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") "
Sep 17 18:57:34.154908 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.154615    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-kserve-provision-location\") pod \"3f343194-783d-45c7-9440-f9cc54889cbb\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") "
Sep 17 18:57:34.154908 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.154652    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-home\") pod \"3f343194-783d-45c7-9440-f9cc54889cbb\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") "
Sep 17 18:57:34.154908 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.154697    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-dshm\") pod \"3f343194-783d-45c7-9440-f9cc54889cbb\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") "
Sep 17 18:57:34.154908 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.154712    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f343194-783d-45c7-9440-f9cc54889cbb" (UID: "3f343194-783d-45c7-9440-f9cc54889cbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:34.154908 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.154757    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2knr9\" (UniqueName: \"kubernetes.io/projected/3f343194-783d-45c7-9440-f9cc54889cbb-kube-api-access-2knr9\") pod \"3f343194-783d-45c7-9440-f9cc54889cbb\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") "
Sep 17 18:57:34.154908 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.154781    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-tmp-dir\") pod \"3f343194-783d-45c7-9440-f9cc54889cbb\" (UID: \"3f343194-783d-45c7-9440-f9cc54889cbb\") "
Sep 17 18:57:34.155375 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.154984    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:57:34.155375 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.155203    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f343194-783d-45c7-9440-f9cc54889cbb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3f343194-783d-45c7-9440-f9cc54889cbb" (UID: "3f343194-783d-45c7-9440-f9cc54889cbb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:34.155490 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.155380    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-home" (OuterVolumeSpecName: "home") pod "3f343194-783d-45c7-9440-f9cc54889cbb" (UID: "3f343194-783d-45c7-9440-f9cc54889cbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:34.158154 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.158125    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f343194-783d-45c7-9440-f9cc54889cbb-kube-api-access-2knr9" (OuterVolumeSpecName: "kube-api-access-2knr9") pod "3f343194-783d-45c7-9440-f9cc54889cbb" (UID: "3f343194-783d-45c7-9440-f9cc54889cbb"). InnerVolumeSpecName "kube-api-access-2knr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:57:34.164278 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.164243    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f343194-783d-45c7-9440-f9cc54889cbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f343194-783d-45c7-9440-f9cc54889cbb" (UID: "3f343194-783d-45c7-9440-f9cc54889cbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:57:34.164490 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.164364    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-dshm" (OuterVolumeSpecName: "dshm") pod "3f343194-783d-45c7-9440-f9cc54889cbb" (UID: "3f343194-783d-45c7-9440-f9cc54889cbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:34.171038 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.171016    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f343194-783d-45c7-9440-f9cc54889cbb" (UID: "3f343194-783d-45c7-9440-f9cc54889cbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:34.182840 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.182773    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f343194-783d-45c7-9440-f9cc54889cbb" (UID: "3f343194-783d-45c7-9440-f9cc54889cbb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:34.255887 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.255855    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:57:34.255887 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.255883    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:57:34.256040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.255900    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2knr9\" (UniqueName: \"kubernetes.io/projected/3f343194-783d-45c7-9440-f9cc54889cbb-kube-api-access-2knr9\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:57:34.256040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.255914    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:57:34.256040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.255926    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f343194-783d-45c7-9440-f9cc54889cbb-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:57:34.256040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.255938    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f343194-783d-45c7-9440-f9cc54889cbb-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:57:34.256040 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.255949    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f343194-783d-45c7-9440-f9cc54889cbb-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:57:34.374150 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.374112    2574 generic.go:356] "Generic (PLEG): container finished" podID="3f343194-783d-45c7-9440-f9cc54889cbb" containerID="623c134936f3d57d6646b5be41ad7b9647b7f1c5e4d873ac222c1c3448f39cac" exitCode=0
Sep 17 18:57:34.374297 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.374162    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"
Sep 17 18:57:34.374297 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.374195    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" event={"ID":"3f343194-783d-45c7-9440-f9cc54889cbb","Type":"ContainerDied","Data":"623c134936f3d57d6646b5be41ad7b9647b7f1c5e4d873ac222c1c3448f39cac"}
Sep 17 18:57:34.374297 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.374265    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj" event={"ID":"3f343194-783d-45c7-9440-f9cc54889cbb","Type":"ContainerDied","Data":"d778d3c5c18c674be0cf75e3430970fe6bef849fae875f666b47e1d454c73cb9"}
Sep 17 18:57:34.374297 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.374294    2574 scope.go:117] "RemoveContainer" containerID="623c134936f3d57d6646b5be41ad7b9647b7f1c5e4d873ac222c1c3448f39cac"
Sep 17 18:57:34.387234 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.387214    2574 scope.go:117] "RemoveContainer" containerID="bb7c9b78205487bca82e648d12926b128bd578bc5bdc6685859945395cc3e366"
Sep 17 18:57:34.396041 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.395935    2574 scope.go:117] "RemoveContainer" containerID="623c134936f3d57d6646b5be41ad7b9647b7f1c5e4d873ac222c1c3448f39cac"
Sep 17 18:57:34.396309 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:57:34.396279    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623c134936f3d57d6646b5be41ad7b9647b7f1c5e4d873ac222c1c3448f39cac\": container with ID starting with 623c134936f3d57d6646b5be41ad7b9647b7f1c5e4d873ac222c1c3448f39cac not found: ID does not exist" containerID="623c134936f3d57d6646b5be41ad7b9647b7f1c5e4d873ac222c1c3448f39cac"
Sep 17 18:57:34.396447 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.396318    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623c134936f3d57d6646b5be41ad7b9647b7f1c5e4d873ac222c1c3448f39cac"} err="failed to get container status \"623c134936f3d57d6646b5be41ad7b9647b7f1c5e4d873ac222c1c3448f39cac\": rpc error: code = NotFound desc = could not find container \"623c134936f3d57d6646b5be41ad7b9647b7f1c5e4d873ac222c1c3448f39cac\": container with ID starting with 623c134936f3d57d6646b5be41ad7b9647b7f1c5e4d873ac222c1c3448f39cac not found: ID does not exist"
Sep 17 18:57:34.396447 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.396342    2574 scope.go:117] "RemoveContainer" containerID="bb7c9b78205487bca82e648d12926b128bd578bc5bdc6685859945395cc3e366"
Sep 17 18:57:34.396762 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:57:34.396639    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7c9b78205487bca82e648d12926b128bd578bc5bdc6685859945395cc3e366\": container with ID starting with bb7c9b78205487bca82e648d12926b128bd578bc5bdc6685859945395cc3e366 not found: ID does not exist" containerID="bb7c9b78205487bca82e648d12926b128bd578bc5bdc6685859945395cc3e366"
Sep 17 18:57:34.396762 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.396673    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7c9b78205487bca82e648d12926b128bd578bc5bdc6685859945395cc3e366"} err="failed to get container status \"bb7c9b78205487bca82e648d12926b128bd578bc5bdc6685859945395cc3e366\": rpc error: code = NotFound desc = could not find container \"bb7c9b78205487bca82e648d12926b128bd578bc5bdc6685859945395cc3e366\": container with ID starting with bb7c9b78205487bca82e648d12926b128bd578bc5bdc6685859945395cc3e366 not found: ID does not exist"
Sep 17 18:57:34.398661 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.398637    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"]
Sep 17 18:57:34.404146 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:34.404128    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6594d79dc8-hzwnj"]
Sep 17 18:57:35.938351 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:57:35.938319    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" path="/var/lib/kubelet/pods/3f343194-783d-45c7-9440-f9cc54889cbb/volumes"
Sep 17 18:58:08.187001 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:08.186961    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:08.283712 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:08.283676    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:14.763754 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:14.763712    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:20.606068 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:20.606033    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:20.709873 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:20.709788    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:21.379733 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:21.379697    2574 scope.go:117] "RemoveContainer" containerID="acbd5c175d48535bc81d067bc47324e1dda123791dd80f0b1ebae396b872da4a"
Sep 17 18:58:22.188993 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.188956    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"]
Sep 17 18:58:22.189547 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.189508    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="storage-initializer"
Sep 17 18:58:22.189663 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.189550    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="storage-initializer"
Sep 17 18:58:22.189663 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.189593    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main"
Sep 17 18:58:22.189663 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.189602    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main"
Sep 17 18:58:22.189833 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.189690    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f343194-783d-45c7-9440-f9cc54889cbb" containerName="main"
Sep 17 18:58:22.194762 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.194743    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.197378 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.197355    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:58:22.197532 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.197474    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 18:58:22.197622 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.197584    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 18:58:22.197622 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.197599    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 18:58:22.197622 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.197606    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 18:58:22.197789 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.197771    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-6clkh\""
Sep 17 18:58:22.205755 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.205733    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"]
Sep 17 18:58:22.226191 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.226170    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:22.269218 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.269193    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-kserve-provision-location\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.269332 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.269224    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-tmp-dir\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.269332 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.269251    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fda5853a-96ba-4dfe-8505-e22a05e5afdc-cabundle-cert\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.269332 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.269312    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-home\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.269462 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.269345    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-model-cache\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.269462 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.269381    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda5853a-96ba-4dfe-8505-e22a05e5afdc-tls-certs\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.269462 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.269436    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cghc8\" (UniqueName: \"kubernetes.io/projected/fda5853a-96ba-4dfe-8505-e22a05e5afdc-kube-api-access-cghc8\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.269462 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.269456    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-dshm\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.370110 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.370072    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-home\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.370313 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.370117    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-model-cache\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.370313 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.370149    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda5853a-96ba-4dfe-8505-e22a05e5afdc-tls-certs\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.370313 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.370187    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cghc8\" (UniqueName: \"kubernetes.io/projected/fda5853a-96ba-4dfe-8505-e22a05e5afdc-kube-api-access-cghc8\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.370313 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.370272    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-dshm\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.370562 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.370325    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-kserve-provision-location\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.370562 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.370360    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-tmp-dir\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.370562 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.370409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fda5853a-96ba-4dfe-8505-e22a05e5afdc-cabundle-cert\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.370562 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.370543    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-home\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.370798 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.370577    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-model-cache\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.370798 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.370673    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-kserve-provision-location\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.370909 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.370841    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-tmp-dir\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.371045 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.371027    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fda5853a-96ba-4dfe-8505-e22a05e5afdc-cabundle-cert\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.372760 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.372742    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-dshm\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.373036 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.373019    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda5853a-96ba-4dfe-8505-e22a05e5afdc-tls-certs\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.380438 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.380415    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cghc8\" (UniqueName: \"kubernetes.io/projected/fda5853a-96ba-4dfe-8505-e22a05e5afdc-kube-api-access-cghc8\") pod \"prestop-hook-test-kserve-59999d87f8-m54lp\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.468224 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.468141    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:22.506571 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.506531    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:22.790372 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:22.790344    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"]
Sep 17 18:58:22.792282 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:58:22.792252    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda5853a_96ba_4dfe_8505_e22a05e5afdc.slice/crio-6342d721f885538057f4aaed503a8df5b74162979466d51eaf0dd964444e65d0 WatchSource:0}: Error finding container 6342d721f885538057f4aaed503a8df5b74162979466d51eaf0dd964444e65d0: Status 404 returned error can't find the container with id 6342d721f885538057f4aaed503a8df5b74162979466d51eaf0dd964444e65d0
Sep 17 18:58:23.655765 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:23.655727    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" event={"ID":"fda5853a-96ba-4dfe-8505-e22a05e5afdc","Type":"ContainerStarted","Data":"b8dd4fdf64ba10c937cff85095cfe32897127f9e3706d0706a6edfdb5dbb8ae3"}
Sep 17 18:58:23.656259 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:23.655823    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" event={"ID":"fda5853a-96ba-4dfe-8505-e22a05e5afdc","Type":"ContainerStarted","Data":"6342d721f885538057f4aaed503a8df5b74162979466d51eaf0dd964444e65d0"}
Sep 17 18:58:25.667610 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:25.667574    2574 generic.go:356] "Generic (PLEG): container finished" podID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerID="b8dd4fdf64ba10c937cff85095cfe32897127f9e3706d0706a6edfdb5dbb8ae3" exitCode=0
Sep 17 18:58:25.668035 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:25.667634    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" event={"ID":"fda5853a-96ba-4dfe-8505-e22a05e5afdc","Type":"ContainerDied","Data":"b8dd4fdf64ba10c937cff85095cfe32897127f9e3706d0706a6edfdb5dbb8ae3"}
Sep 17 18:58:26.675971 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:26.675930    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" event={"ID":"fda5853a-96ba-4dfe-8505-e22a05e5afdc","Type":"ContainerStarted","Data":"4eff512aa2370c1cc503d69331ef4ba97daf4cca761edcb76ebc4ffa437cbd50"}
Sep 17 18:58:26.700632 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:26.700571    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" podStartSLOduration=2.431438961 podStartE2EDuration="4.700554757s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.269115796s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:58:26.697001748 +0000 UTC m=+3676.997824063" watchObservedRunningTime="2026-09-17 18:58:26.700554757 +0000 UTC m=+3677.001377075"
Sep 17 18:58:27.507609 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:27.507566    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:28.598888 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:28.598839    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:28.895327 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:28.895228    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:32.507508 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:32.507466    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 18:58:32.509237 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:32.509206    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Sep 17 18:58:36.281629 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:36.281590    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:36.381527 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:36.381476    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:42.507028 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:42.506986    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Sep 17 18:58:52.507336 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:52.507283    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Sep 17 18:58:52.554256 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:58:52.554219    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:59:02.507269 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:02.507221    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Sep 17 18:59:07.577476 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:07.577409    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="d306c934-15fd-4b88-b641-ffe2aa31049b" containerName="main" containerID="cri-o://e6b41e7cf7597593cdc8835f84ed06c8697f63f133bda1024f957722ce979b5f" gracePeriod=45
Sep 17 18:59:08.243735 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.243708    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:59:08.286656 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.286621    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr5kn\" (UniqueName: \"kubernetes.io/projected/d306c934-15fd-4b88-b641-ffe2aa31049b-kube-api-access-fr5kn\") pod \"d306c934-15fd-4b88-b641-ffe2aa31049b\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") "
Sep 17 18:59:08.286836 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.286668    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d306c934-15fd-4b88-b641-ffe2aa31049b-cabundle-cert\") pod \"d306c934-15fd-4b88-b641-ffe2aa31049b\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") "
Sep 17 18:59:08.286836 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.286710    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-tmp-dir\") pod \"d306c934-15fd-4b88-b641-ffe2aa31049b\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") "
Sep 17 18:59:08.286836 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.286750    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-dshm\") pod \"d306c934-15fd-4b88-b641-ffe2aa31049b\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") "
Sep 17 18:59:08.286836 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.286774    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-home\") pod \"d306c934-15fd-4b88-b641-ffe2aa31049b\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") "
Sep 17 18:59:08.286836 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.286802    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d306c934-15fd-4b88-b641-ffe2aa31049b-tls-certs\") pod \"d306c934-15fd-4b88-b641-ffe2aa31049b\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") "
Sep 17 18:59:08.286836 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.286836    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-kserve-provision-location\") pod \"d306c934-15fd-4b88-b641-ffe2aa31049b\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") "
Sep 17 18:59:08.287151 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.286863    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-model-cache\") pod \"d306c934-15fd-4b88-b641-ffe2aa31049b\" (UID: \"d306c934-15fd-4b88-b641-ffe2aa31049b\") "
Sep 17 18:59:08.287206 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.287144    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d306c934-15fd-4b88-b641-ffe2aa31049b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d306c934-15fd-4b88-b641-ffe2aa31049b" (UID: "d306c934-15fd-4b88-b641-ffe2aa31049b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:59:08.287357 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.287308    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d306c934-15fd-4b88-b641-ffe2aa31049b" (UID: "d306c934-15fd-4b88-b641-ffe2aa31049b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:08.288226 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.288198    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-home" (OuterVolumeSpecName: "home") pod "d306c934-15fd-4b88-b641-ffe2aa31049b" (UID: "d306c934-15fd-4b88-b641-ffe2aa31049b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:08.289813 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.289772    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d306c934-15fd-4b88-b641-ffe2aa31049b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d306c934-15fd-4b88-b641-ffe2aa31049b" (UID: "d306c934-15fd-4b88-b641-ffe2aa31049b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:59:08.290089 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.290054    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d306c934-15fd-4b88-b641-ffe2aa31049b-kube-api-access-fr5kn" (OuterVolumeSpecName: "kube-api-access-fr5kn") pod "d306c934-15fd-4b88-b641-ffe2aa31049b" (UID: "d306c934-15fd-4b88-b641-ffe2aa31049b"). InnerVolumeSpecName "kube-api-access-fr5kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:59:08.290195 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.290178    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-dshm" (OuterVolumeSpecName: "dshm") pod "d306c934-15fd-4b88-b641-ffe2aa31049b" (UID: "d306c934-15fd-4b88-b641-ffe2aa31049b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:08.302421 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.302383    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d306c934-15fd-4b88-b641-ffe2aa31049b" (UID: "d306c934-15fd-4b88-b641-ffe2aa31049b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:08.313038 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.313011    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d306c934-15fd-4b88-b641-ffe2aa31049b" (UID: "d306c934-15fd-4b88-b641-ffe2aa31049b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:59:08.388346 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.388247    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d306c934-15fd-4b88-b641-ffe2aa31049b-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:59:08.388346 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.388280    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:59:08.388346 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.388293    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:59:08.388346 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.388305    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr5kn\" (UniqueName: \"kubernetes.io/projected/d306c934-15fd-4b88-b641-ffe2aa31049b-kube-api-access-fr5kn\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:59:08.388346 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.388315    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d306c934-15fd-4b88-b641-ffe2aa31049b-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:59:08.388648 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.388353    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:59:08.388648 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.388365    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:59:08.388648 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.388374    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d306c934-15fd-4b88-b641-ffe2aa31049b-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 18:59:08.921630 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.921584    2574 generic.go:356] "Generic (PLEG): container finished" podID="d306c934-15fd-4b88-b641-ffe2aa31049b" containerID="e6b41e7cf7597593cdc8835f84ed06c8697f63f133bda1024f957722ce979b5f" exitCode=0
Sep 17 18:59:08.922061 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.921644    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:59:08.922061 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.921670    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d306c934-15fd-4b88-b641-ffe2aa31049b","Type":"ContainerDied","Data":"e6b41e7cf7597593cdc8835f84ed06c8697f63f133bda1024f957722ce979b5f"}
Sep 17 18:59:08.922061 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.921732    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d306c934-15fd-4b88-b641-ffe2aa31049b","Type":"ContainerDied","Data":"bf12adba5294bf23bdf988f8a2b05c96abf356f5b2c64be21062cb78df530d13"}
Sep 17 18:59:08.922061 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.921762    2574 scope.go:117] "RemoveContainer" containerID="e6b41e7cf7597593cdc8835f84ed06c8697f63f133bda1024f957722ce979b5f"
Sep 17 18:59:08.933127 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.933106    2574 scope.go:117] "RemoveContainer" containerID="7c7dbade9a968c6bdcabf53037ae55c726a270ceb847755b7d21ac586c0bdf7d"
Sep 17 18:59:08.941064 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.941044    2574 scope.go:117] "RemoveContainer" containerID="e6b41e7cf7597593cdc8835f84ed06c8697f63f133bda1024f957722ce979b5f"
Sep 17 18:59:08.941400 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:59:08.941366    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b41e7cf7597593cdc8835f84ed06c8697f63f133bda1024f957722ce979b5f\": container with ID starting with e6b41e7cf7597593cdc8835f84ed06c8697f63f133bda1024f957722ce979b5f not found: ID does not exist" containerID="e6b41e7cf7597593cdc8835f84ed06c8697f63f133bda1024f957722ce979b5f"
Sep 17 18:59:08.941565 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.941403    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b41e7cf7597593cdc8835f84ed06c8697f63f133bda1024f957722ce979b5f"} err="failed to get container status \"e6b41e7cf7597593cdc8835f84ed06c8697f63f133bda1024f957722ce979b5f\": rpc error: code = NotFound desc = could not find container \"e6b41e7cf7597593cdc8835f84ed06c8697f63f133bda1024f957722ce979b5f\": container with ID starting with e6b41e7cf7597593cdc8835f84ed06c8697f63f133bda1024f957722ce979b5f not found: ID does not exist"
Sep 17 18:59:08.941565 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.941426    2574 scope.go:117] "RemoveContainer" containerID="7c7dbade9a968c6bdcabf53037ae55c726a270ceb847755b7d21ac586c0bdf7d"
Sep 17 18:59:08.941819 ip-10-0-132-0 kubenswrapper[2574]: E0917 18:59:08.941789    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7dbade9a968c6bdcabf53037ae55c726a270ceb847755b7d21ac586c0bdf7d\": container with ID starting with 7c7dbade9a968c6bdcabf53037ae55c726a270ceb847755b7d21ac586c0bdf7d not found: ID does not exist" containerID="7c7dbade9a968c6bdcabf53037ae55c726a270ceb847755b7d21ac586c0bdf7d"
Sep 17 18:59:08.941931 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.941826    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7dbade9a968c6bdcabf53037ae55c726a270ceb847755b7d21ac586c0bdf7d"} err="failed to get container status \"7c7dbade9a968c6bdcabf53037ae55c726a270ceb847755b7d21ac586c0bdf7d\": rpc error: code = NotFound desc = could not find container \"7c7dbade9a968c6bdcabf53037ae55c726a270ceb847755b7d21ac586c0bdf7d\": container with ID starting with 7c7dbade9a968c6bdcabf53037ae55c726a270ceb847755b7d21ac586c0bdf7d not found: ID does not exist"
Sep 17 18:59:08.946048 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.946016    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:59:08.949540 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:08.949501    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:59:09.938501 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:09.938465    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d306c934-15fd-4b88-b641-ffe2aa31049b" path="/var/lib/kubelet/pods/d306c934-15fd-4b88-b641-ffe2aa31049b/volumes"
Sep 17 18:59:12.507506 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:12.507456    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Sep 17 18:59:22.507223 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:22.507182    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Sep 17 18:59:26.380978 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:26.380910    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:26.480556 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:26.480504    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:32.507281 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:32.507236    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Sep 17 18:59:33.083776 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:33.083734    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:38.963260 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:38.963217    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:39.079953 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:39.079909    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:40.509418 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.509378    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:40.778014 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.777918    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"]
Sep 17 18:59:40.778614 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.778585    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d306c934-15fd-4b88-b641-ffe2aa31049b" containerName="storage-initializer"
Sep 17 18:59:40.778739 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.778615    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d306c934-15fd-4b88-b641-ffe2aa31049b" containerName="storage-initializer"
Sep 17 18:59:40.778739 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.778627    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d306c934-15fd-4b88-b641-ffe2aa31049b" containerName="main"
Sep 17 18:59:40.778739 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.778635    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d306c934-15fd-4b88-b641-ffe2aa31049b" containerName="main"
Sep 17 18:59:40.778739 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.778735    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d306c934-15fd-4b88-b641-ffe2aa31049b" containerName="main"
Sep 17 18:59:40.783752 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.783723    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"
Sep 17 18:59:40.787189 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.787159    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-vgbhd\""
Sep 17 18:59:40.787352 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.787326    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 18:59:40.787420 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.787346    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 18:59:40.787814 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.787791    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 18:59:40.787942 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.787858    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-p6fxs\""
Sep 17 18:59:40.791317 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.791286    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"]
Sep 17 18:59:40.809565 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.809529    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:40.866749 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.866708    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzhd4\" (UniqueName: \"kubernetes.io/projected/9feb15b3-4b42-4a87-8204-84d6b8105993-kube-api-access-nzhd4\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn\" (UID: \"9feb15b3-4b42-4a87-8204-84d6b8105993\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"
Sep 17 18:59:40.866927 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.866784    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9feb15b3-4b42-4a87-8204-84d6b8105993-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn\" (UID: \"9feb15b3-4b42-4a87-8204-84d6b8105993\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"
Sep 17 18:59:40.967508 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.967473    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzhd4\" (UniqueName: \"kubernetes.io/projected/9feb15b3-4b42-4a87-8204-84d6b8105993-kube-api-access-nzhd4\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn\" (UID: \"9feb15b3-4b42-4a87-8204-84d6b8105993\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"
Sep 17 18:59:40.967711 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.967533    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9feb15b3-4b42-4a87-8204-84d6b8105993-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn\" (UID: \"9feb15b3-4b42-4a87-8204-84d6b8105993\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"
Sep 17 18:59:40.970207 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.970178    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9feb15b3-4b42-4a87-8204-84d6b8105993-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn\" (UID: \"9feb15b3-4b42-4a87-8204-84d6b8105993\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"
Sep 17 18:59:40.976220 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:40.976194    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzhd4\" (UniqueName: \"kubernetes.io/projected/9feb15b3-4b42-4a87-8204-84d6b8105993-kube-api-access-nzhd4\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn\" (UID: \"9feb15b3-4b42-4a87-8204-84d6b8105993\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"
Sep 17 18:59:41.095961 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:41.095883    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"
Sep 17 18:59:41.404724 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:41.404689    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"]
Sep 17 18:59:41.405166 ip-10-0-132-0 kubenswrapper[2574]: W0917 18:59:41.405136    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9feb15b3_4b42_4a87_8204_84d6b8105993.slice/crio-5698830d1cc3bac7042520b0c266a591c5ac625260f0ec1811d767c777f14741 WatchSource:0}: Error finding container 5698830d1cc3bac7042520b0c266a591c5ac625260f0ec1811d767c777f14741: Status 404 returned error can't find the container with id 5698830d1cc3bac7042520b0c266a591c5ac625260f0ec1811d767c777f14741
Sep 17 18:59:42.117025 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:42.116985    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn" event={"ID":"9feb15b3-4b42-4a87-8204-84d6b8105993","Type":"ContainerStarted","Data":"4787f746e8eb658acc1dcda8b972fe29591bdcfbb7deb0e87c9f7e766e855808"}
Sep 17 18:59:42.117436 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:42.117044    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn" event={"ID":"9feb15b3-4b42-4a87-8204-84d6b8105993","Type":"ContainerStarted","Data":"5698830d1cc3bac7042520b0c266a591c5ac625260f0ec1811d767c777f14741"}
Sep 17 18:59:42.117436 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:42.117072    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"
Sep 17 18:59:42.136428 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:42.136378    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn" podStartSLOduration=2.13636318 podStartE2EDuration="2.13636318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:59:42.134804319 +0000 UTC m=+3752.435626644" watchObservedRunningTime="2026-09-17 18:59:42.13636318 +0000 UTC m=+3752.437185531"
Sep 17 18:59:42.507677 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:42.507640    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Sep 17 18:59:52.507992 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:52.507890    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Sep 17 18:59:58.620714 ip-10-0-132-0 kubenswrapper[2574]: I0917 18:59:58.620679    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:02.507312 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:02.507261    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Sep 17 19:00:12.507649 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:12.507603    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Sep 17 19:00:13.124386 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:13.124357    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"
Sep 17 19:00:14.603705 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:14.603667    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:14.606052 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:14.605750    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"]
Sep 17 19:00:14.704095 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:14.704063    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:22.517219 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:22.517187    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 19:00:22.525021 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:22.524996    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 19:00:23.124915 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:23.124877    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn" podUID="9feb15b3-4b42-4a87-8204-84d6b8105993" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:00:24.016766 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:24.016714    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"]
Sep 17 19:00:29.630110 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:29.630074    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn" podUID="9feb15b3-4b42-4a87-8204-84d6b8105993" containerName="main" containerID="cri-o://4787f746e8eb658acc1dcda8b972fe29591bdcfbb7deb0e87c9f7e766e855808" gracePeriod=45
Sep 17 19:00:33.124170 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:33.124126    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn" podUID="9feb15b3-4b42-4a87-8204-84d6b8105993" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:00:39.372802 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:39.372753    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main" containerID="cri-o://4eff512aa2370c1cc503d69331ef4ba97daf4cca761edcb76ebc4ffa437cbd50" gracePeriod=45
Sep 17 19:00:39.518393 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:39.518354    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Sep 17 19:00:40.050374 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.050345    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 19:00:40.173486 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.173404    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda5853a-96ba-4dfe-8505-e22a05e5afdc-tls-certs\") pod \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") "
Sep 17 19:00:40.173486 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.173458    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-home\") pod \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") "
Sep 17 19:00:40.173733 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.173507    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-kserve-provision-location\") pod \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") "
Sep 17 19:00:40.173733 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.173547    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cghc8\" (UniqueName: \"kubernetes.io/projected/fda5853a-96ba-4dfe-8505-e22a05e5afdc-kube-api-access-cghc8\") pod \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") "
Sep 17 19:00:40.173733 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.173566    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-model-cache\") pod \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") "
Sep 17 19:00:40.173733 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.173587    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-dshm\") pod \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") "
Sep 17 19:00:40.173733 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.173613    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fda5853a-96ba-4dfe-8505-e22a05e5afdc-cabundle-cert\") pod \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") "
Sep 17 19:00:40.173733 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.173642    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-tmp-dir\") pod \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\" (UID: \"fda5853a-96ba-4dfe-8505-e22a05e5afdc\") "
Sep 17 19:00:40.174030 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.173818    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-model-cache" (OuterVolumeSpecName: "model-cache") pod "fda5853a-96ba-4dfe-8505-e22a05e5afdc" (UID: "fda5853a-96ba-4dfe-8505-e22a05e5afdc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:40.174250 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.174224    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-home" (OuterVolumeSpecName: "home") pod "fda5853a-96ba-4dfe-8505-e22a05e5afdc" (UID: "fda5853a-96ba-4dfe-8505-e22a05e5afdc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:40.174634 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.174601    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda5853a-96ba-4dfe-8505-e22a05e5afdc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fda5853a-96ba-4dfe-8505-e22a05e5afdc" (UID: "fda5853a-96ba-4dfe-8505-e22a05e5afdc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:00:40.175999 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.175969    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda5853a-96ba-4dfe-8505-e22a05e5afdc-kube-api-access-cghc8" (OuterVolumeSpecName: "kube-api-access-cghc8") pod "fda5853a-96ba-4dfe-8505-e22a05e5afdc" (UID: "fda5853a-96ba-4dfe-8505-e22a05e5afdc"). InnerVolumeSpecName "kube-api-access-cghc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:00:40.176166 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.176144    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda5853a-96ba-4dfe-8505-e22a05e5afdc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fda5853a-96ba-4dfe-8505-e22a05e5afdc" (UID: "fda5853a-96ba-4dfe-8505-e22a05e5afdc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:00:40.176319 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.176293    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-dshm" (OuterVolumeSpecName: "dshm") pod "fda5853a-96ba-4dfe-8505-e22a05e5afdc" (UID: "fda5853a-96ba-4dfe-8505-e22a05e5afdc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:40.187558 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.187531    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fda5853a-96ba-4dfe-8505-e22a05e5afdc" (UID: "fda5853a-96ba-4dfe-8505-e22a05e5afdc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:40.204857 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.204830    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fda5853a-96ba-4dfe-8505-e22a05e5afdc" (UID: "fda5853a-96ba-4dfe-8505-e22a05e5afdc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:40.274732 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.274708    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda5853a-96ba-4dfe-8505-e22a05e5afdc-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:00:40.274732 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.274727    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:00:40.274881 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.274737    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-kserve-provision-location\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:00:40.274881 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.274747    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cghc8\" (UniqueName: \"kubernetes.io/projected/fda5853a-96ba-4dfe-8505-e22a05e5afdc-kube-api-access-cghc8\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:00:40.274881 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.274755    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:00:40.274881 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.274763    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:00:40.274881 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.274770    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fda5853a-96ba-4dfe-8505-e22a05e5afdc-cabundle-cert\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:00:40.274881 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.274777    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda5853a-96ba-4dfe-8505-e22a05e5afdc-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:00:40.436928 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.436865    2574 generic.go:356] "Generic (PLEG): container finished" podID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerID="4eff512aa2370c1cc503d69331ef4ba97daf4cca761edcb76ebc4ffa437cbd50" exitCode=0
Sep 17 19:00:40.436928 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.436916    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"
Sep 17 19:00:40.437370 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.436947    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" event={"ID":"fda5853a-96ba-4dfe-8505-e22a05e5afdc","Type":"ContainerDied","Data":"4eff512aa2370c1cc503d69331ef4ba97daf4cca761edcb76ebc4ffa437cbd50"}
Sep 17 19:00:40.437370 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.437014    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp" event={"ID":"fda5853a-96ba-4dfe-8505-e22a05e5afdc","Type":"ContainerDied","Data":"6342d721f885538057f4aaed503a8df5b74162979466d51eaf0dd964444e65d0"}
Sep 17 19:00:40.437370 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.437051    2574 scope.go:117] "RemoveContainer" containerID="4eff512aa2370c1cc503d69331ef4ba97daf4cca761edcb76ebc4ffa437cbd50"
Sep 17 19:00:40.446483 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.446466    2574 scope.go:117] "RemoveContainer" containerID="b8dd4fdf64ba10c937cff85095cfe32897127f9e3706d0706a6edfdb5dbb8ae3"
Sep 17 19:00:40.457917 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.457893    2574 scope.go:117] "RemoveContainer" containerID="4eff512aa2370c1cc503d69331ef4ba97daf4cca761edcb76ebc4ffa437cbd50"
Sep 17 19:00:40.458276 ip-10-0-132-0 kubenswrapper[2574]: E0917 19:00:40.458189    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eff512aa2370c1cc503d69331ef4ba97daf4cca761edcb76ebc4ffa437cbd50\": container with ID starting with 4eff512aa2370c1cc503d69331ef4ba97daf4cca761edcb76ebc4ffa437cbd50 not found: ID does not exist" containerID="4eff512aa2370c1cc503d69331ef4ba97daf4cca761edcb76ebc4ffa437cbd50"
Sep 17 19:00:40.458367 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.458273    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eff512aa2370c1cc503d69331ef4ba97daf4cca761edcb76ebc4ffa437cbd50"} err="failed to get container status \"4eff512aa2370c1cc503d69331ef4ba97daf4cca761edcb76ebc4ffa437cbd50\": rpc error: code = NotFound desc = could not find container \"4eff512aa2370c1cc503d69331ef4ba97daf4cca761edcb76ebc4ffa437cbd50\": container with ID starting with 4eff512aa2370c1cc503d69331ef4ba97daf4cca761edcb76ebc4ffa437cbd50 not found: ID does not exist"
Sep 17 19:00:40.458367 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.458297    2574 scope.go:117] "RemoveContainer" containerID="b8dd4fdf64ba10c937cff85095cfe32897127f9e3706d0706a6edfdb5dbb8ae3"
Sep 17 19:00:40.458758 ip-10-0-132-0 kubenswrapper[2574]: E0917 19:00:40.458655    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8dd4fdf64ba10c937cff85095cfe32897127f9e3706d0706a6edfdb5dbb8ae3\": container with ID starting with b8dd4fdf64ba10c937cff85095cfe32897127f9e3706d0706a6edfdb5dbb8ae3 not found: ID does not exist" containerID="b8dd4fdf64ba10c937cff85095cfe32897127f9e3706d0706a6edfdb5dbb8ae3"
Sep 17 19:00:40.458758 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.458690    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8dd4fdf64ba10c937cff85095cfe32897127f9e3706d0706a6edfdb5dbb8ae3"} err="failed to get container status \"b8dd4fdf64ba10c937cff85095cfe32897127f9e3706d0706a6edfdb5dbb8ae3\": rpc error: code = NotFound desc = could not find container \"b8dd4fdf64ba10c937cff85095cfe32897127f9e3706d0706a6edfdb5dbb8ae3\": container with ID starting with b8dd4fdf64ba10c937cff85095cfe32897127f9e3706d0706a6edfdb5dbb8ae3 not found: ID does not exist"
Sep 17 19:00:40.460928 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.460904    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"]
Sep 17 19:00:40.464195 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:40.464175    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-m54lp"]
Sep 17 19:00:41.939374 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:41.939339    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" path="/var/lib/kubelet/pods/fda5853a-96ba-4dfe-8505-e22a05e5afdc/volumes"
Sep 17 19:00:43.124238 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:43.124206    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn" podUID="9feb15b3-4b42-4a87-8204-84d6b8105993" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:00:43.124659 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:43.124320    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"
Sep 17 19:00:52.581036 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:52.580999    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:52.674746 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:52.674711    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:53.124885 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:53.124845    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn" podUID="9feb15b3-4b42-4a87-8204-84d6b8105993" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:00:59.776161 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:59.776119    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:59.884766 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:00:59.884739    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"
Sep 17 19:01:00.024426 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.024396    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzhd4\" (UniqueName: \"kubernetes.io/projected/9feb15b3-4b42-4a87-8204-84d6b8105993-kube-api-access-nzhd4\") pod \"9feb15b3-4b42-4a87-8204-84d6b8105993\" (UID: \"9feb15b3-4b42-4a87-8204-84d6b8105993\") "
Sep 17 19:01:00.024632 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.024530    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9feb15b3-4b42-4a87-8204-84d6b8105993-tls-certs\") pod \"9feb15b3-4b42-4a87-8204-84d6b8105993\" (UID: \"9feb15b3-4b42-4a87-8204-84d6b8105993\") "
Sep 17 19:01:00.026772 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.026744    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9feb15b3-4b42-4a87-8204-84d6b8105993-kube-api-access-nzhd4" (OuterVolumeSpecName: "kube-api-access-nzhd4") pod "9feb15b3-4b42-4a87-8204-84d6b8105993" (UID: "9feb15b3-4b42-4a87-8204-84d6b8105993"). InnerVolumeSpecName "kube-api-access-nzhd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:01:00.026892 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.026846    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9feb15b3-4b42-4a87-8204-84d6b8105993-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9feb15b3-4b42-4a87-8204-84d6b8105993" (UID: "9feb15b3-4b42-4a87-8204-84d6b8105993"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:01:00.125819 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.125781    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9feb15b3-4b42-4a87-8204-84d6b8105993-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:01:00.125819 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.125809    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzhd4\" (UniqueName: \"kubernetes.io/projected/9feb15b3-4b42-4a87-8204-84d6b8105993-kube-api-access-nzhd4\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:01:00.556149 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.556110    2574 generic.go:356] "Generic (PLEG): container finished" podID="9feb15b3-4b42-4a87-8204-84d6b8105993" containerID="4787f746e8eb658acc1dcda8b972fe29591bdcfbb7deb0e87c9f7e766e855808" exitCode=0
Sep 17 19:01:00.556351 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.556167    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"
Sep 17 19:01:00.556351 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.556210    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn" event={"ID":"9feb15b3-4b42-4a87-8204-84d6b8105993","Type":"ContainerDied","Data":"4787f746e8eb658acc1dcda8b972fe29591bdcfbb7deb0e87c9f7e766e855808"}
Sep 17 19:01:00.556351 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.556282    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn" event={"ID":"9feb15b3-4b42-4a87-8204-84d6b8105993","Type":"ContainerDied","Data":"5698830d1cc3bac7042520b0c266a591c5ac625260f0ec1811d767c777f14741"}
Sep 17 19:01:00.556351 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.556304    2574 scope.go:117] "RemoveContainer" containerID="4787f746e8eb658acc1dcda8b972fe29591bdcfbb7deb0e87c9f7e766e855808"
Sep 17 19:01:00.567457 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.567443    2574 scope.go:117] "RemoveContainer" containerID="4787f746e8eb658acc1dcda8b972fe29591bdcfbb7deb0e87c9f7e766e855808"
Sep 17 19:01:00.567735 ip-10-0-132-0 kubenswrapper[2574]: E0917 19:01:00.567716    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4787f746e8eb658acc1dcda8b972fe29591bdcfbb7deb0e87c9f7e766e855808\": container with ID starting with 4787f746e8eb658acc1dcda8b972fe29591bdcfbb7deb0e87c9f7e766e855808 not found: ID does not exist" containerID="4787f746e8eb658acc1dcda8b972fe29591bdcfbb7deb0e87c9f7e766e855808"
Sep 17 19:01:00.567793 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.567742    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4787f746e8eb658acc1dcda8b972fe29591bdcfbb7deb0e87c9f7e766e855808"} err="failed to get container status \"4787f746e8eb658acc1dcda8b972fe29591bdcfbb7deb0e87c9f7e766e855808\": rpc error: code = NotFound desc = could not find container \"4787f746e8eb658acc1dcda8b972fe29591bdcfbb7deb0e87c9f7e766e855808\": container with ID starting with 4787f746e8eb658acc1dcda8b972fe29591bdcfbb7deb0e87c9f7e766e855808 not found: ID does not exist"
Sep 17 19:01:00.578767 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.578741    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"]
Sep 17 19:01:00.584937 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:00.584917    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbkw2nn"]
Sep 17 19:01:01.938211 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:01.938177    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9feb15b3-4b42-4a87-8204-84d6b8105993" path="/var/lib/kubelet/pods/9feb15b3-4b42-4a87-8204-84d6b8105993/volumes"
Sep 17 19:01:04.693374 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:04.693338    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:04.792721 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:04.792685    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:10.898536 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:10.898491    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:16.936400 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:16.936345    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:17.054314 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:17.054271    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:23.380347 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.380311    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"]
Sep 17 19:01:23.380749 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.380671    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main"
Sep 17 19:01:23.380749 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.380683    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main"
Sep 17 19:01:23.380749 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.380701    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9feb15b3-4b42-4a87-8204-84d6b8105993" containerName="main"
Sep 17 19:01:23.380749 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.380706    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9feb15b3-4b42-4a87-8204-84d6b8105993" containerName="main"
Sep 17 19:01:23.380749 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.380717    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="storage-initializer"
Sep 17 19:01:23.380749 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.380722    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="storage-initializer"
Sep 17 19:01:23.380953 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.380781    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9feb15b3-4b42-4a87-8204-84d6b8105993" containerName="main"
Sep 17 19:01:23.380953 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.380790    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda5853a-96ba-4dfe-8505-e22a05e5afdc" containerName="main"
Sep 17 19:01:23.384999 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.384981    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.387328 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.387308    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 19:01:23.388752 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.388718    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 19:01:23.390574 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.389003    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-6xcb7\""
Sep 17 19:01:23.390574 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.389998    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 19:01:23.398197 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.398174    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"]
Sep 17 19:01:23.410144 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.410115    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:23.410472 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.410450    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722d209e-1bc2-4607-87c4-ac89404a1548-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.410573 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.410495    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.410661 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.410634    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.410712 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.410678    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sb4j\" (UniqueName: \"kubernetes.io/projected/722d209e-1bc2-4607-87c4-ac89404a1548-kube-api-access-5sb4j\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.410767 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.410742    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.410854 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.410823    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.511489 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.511446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.511489 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.511495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.511729 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.511543    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722d209e-1bc2-4607-87c4-ac89404a1548-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.511729 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.511566    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.511729 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.511609    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.511729 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.511633    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sb4j\" (UniqueName: \"kubernetes.io/projected/722d209e-1bc2-4607-87c4-ac89404a1548-kube-api-access-5sb4j\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.511943 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.511921    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.512008 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.511927    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.512072 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.512052    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.514201 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.514182    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.514325 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.514288    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722d209e-1bc2-4607-87c4-ac89404a1548-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.520889 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.520865    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sb4j\" (UniqueName: \"kubernetes.io/projected/722d209e-1bc2-4607-87c4-ac89404a1548-kube-api-access-5sb4j\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.686015 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.685932    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:23.701101 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.701069    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:23.975698 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:23.975673    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"]
Sep 17 19:01:23.978137 ip-10-0-132-0 kubenswrapper[2574]: W0917 19:01:23.978105    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722d209e_1bc2_4607_87c4_ac89404a1548.slice/crio-42e7fd67e8394bd2da74d07603c2be4d538602e03914962356cbc3e4bcc0d3bd WatchSource:0}: Error finding container 42e7fd67e8394bd2da74d07603c2be4d538602e03914962356cbc3e4bcc0d3bd: Status 404 returned error can't find the container with id 42e7fd67e8394bd2da74d07603c2be4d538602e03914962356cbc3e4bcc0d3bd
Sep 17 19:01:24.688847 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:24.688806    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722" event={"ID":"722d209e-1bc2-4607-87c4-ac89404a1548","Type":"ContainerStarted","Data":"16356e7b7163f0d64ba1f9b2414859c1efcd0e7c6285b2a1ab7b2adcca159fbb"}
Sep 17 19:01:24.689237 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:24.688862    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722" event={"ID":"722d209e-1bc2-4607-87c4-ac89404a1548","Type":"ContainerStarted","Data":"42e7fd67e8394bd2da74d07603c2be4d538602e03914962356cbc3e4bcc0d3bd"}
Sep 17 19:01:24.701664 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:24.701636    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:24.711805 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:24.711754    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722" podStartSLOduration=1.711739616 podStartE2EDuration="1.711739616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:01:24.710334835 +0000 UTC m=+3855.011157161" watchObservedRunningTime="2026-09-17 19:01:24.711739616 +0000 UTC m=+3855.012561939"
Sep 17 19:01:32.664337 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:32.664300    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:33.701603 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:33.701565    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:33.714201 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:33.714179    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:33.748058 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:33.748032    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:01:42.611306 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:42.611273    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:42.708669 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:42.708638    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:49.162533 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:49.162484    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:51.209842 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.209803    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:51.423872 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.423831    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"]
Sep 17 19:01:51.427886 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.427863    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"
Sep 17 19:01:51.431653 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.431573    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-8zwt9\""
Sep 17 19:01:51.431928 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.431897    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-rpptv\""
Sep 17 19:01:51.432113 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.432086    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 19:01:51.432235 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.432179    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 19:01:51.432235 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.431904    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 19:01:51.434792 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.434765    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"]
Sep 17 19:01:51.480988 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.480921    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:51.537135 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.537111    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkfps\" (UniqueName: \"kubernetes.io/projected/8c88884f-712e-4dcf-9aba-833df3d6259b-kube-api-access-gkfps\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl\" (UID: \"8c88884f-712e-4dcf-9aba-833df3d6259b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"
Sep 17 19:01:51.537242 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.537173    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c88884f-712e-4dcf-9aba-833df3d6259b-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl\" (UID: \"8c88884f-712e-4dcf-9aba-833df3d6259b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"
Sep 17 19:01:51.638439 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.638402    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkfps\" (UniqueName: \"kubernetes.io/projected/8c88884f-712e-4dcf-9aba-833df3d6259b-kube-api-access-gkfps\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl\" (UID: \"8c88884f-712e-4dcf-9aba-833df3d6259b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"
Sep 17 19:01:51.638618 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.638493    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c88884f-712e-4dcf-9aba-833df3d6259b-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl\" (UID: \"8c88884f-712e-4dcf-9aba-833df3d6259b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"
Sep 17 19:01:51.641364 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.641344    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c88884f-712e-4dcf-9aba-833df3d6259b-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl\" (UID: \"8c88884f-712e-4dcf-9aba-833df3d6259b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"
Sep 17 19:01:51.646884 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.646855    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkfps\" (UniqueName: \"kubernetes.io/projected/8c88884f-712e-4dcf-9aba-833df3d6259b-kube-api-access-gkfps\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl\" (UID: \"8c88884f-712e-4dcf-9aba-833df3d6259b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"
Sep 17 19:01:51.739934 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:51.739876    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"
Sep 17 19:01:52.017080 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:52.017052    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"]
Sep 17 19:01:52.018973 ip-10-0-132-0 kubenswrapper[2574]: W0917 19:01:52.018946    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c88884f_712e_4dcf_9aba_833df3d6259b.slice/crio-35b5714b9d7678f659e30fcbdc94241f03958b1e8df5187e0d2a257fc3868ad8 WatchSource:0}: Error finding container 35b5714b9d7678f659e30fcbdc94241f03958b1e8df5187e0d2a257fc3868ad8: Status 404 returned error can't find the container with id 35b5714b9d7678f659e30fcbdc94241f03958b1e8df5187e0d2a257fc3868ad8
Sep 17 19:01:52.837452 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:52.837410    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl" event={"ID":"8c88884f-712e-4dcf-9aba-833df3d6259b","Type":"ContainerStarted","Data":"3031a49fc81305e2213e6c5dc8cf8b894770c69f13f0a3a25a1c453586742cac"}
Sep 17 19:01:52.837877 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:52.837571    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"
Sep 17 19:01:52.837877 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:52.837612    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl" event={"ID":"8c88884f-712e-4dcf-9aba-833df3d6259b","Type":"ContainerStarted","Data":"35b5714b9d7678f659e30fcbdc94241f03958b1e8df5187e0d2a257fc3868ad8"}
Sep 17 19:01:52.854402 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:52.854356    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl" podStartSLOduration=1.854343082 podStartE2EDuration="1.854343082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:01:52.852893833 +0000 UTC m=+3883.153716157" watchObservedRunningTime="2026-09-17 19:01:52.854343082 +0000 UTC m=+3883.155165432"
Sep 17 19:01:55.554973 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:55.554927    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:55.631601 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:01:55.631574    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:05.276683 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:05.276637    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:08.208718 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:08.208679    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:08.227128 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:08.227099    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"]
Sep 17 19:02:08.235921 ip-10-0-132-0 kubenswrapper[2574]: E0917 19:02:08.235886    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:02:08.235921 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:08.235921 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:08.235921 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="16356e7b7163f0d64ba1f9b2414859c1efcd0e7c6285b2a1ab7b2adcca159fbb" cmd=["/bin/sleep","15"]
Sep 17 19:02:08.236180 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:08.235935    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722" message="" err=<
Sep 17 19:02:08.236180 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:08.236180 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:08.236180 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 19:02:08.236180 ip-10-0-132-0 kubenswrapper[2574]: E0917 19:02:08.235947    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:02:08.236180 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:08.236180 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:08.236180 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722" podUID="722d209e-1bc2-4607-87c4-ac89404a1548" containerName="main" containerID="cri-o://16356e7b7163f0d64ba1f9b2414859c1efcd0e7c6285b2a1ab7b2adcca159fbb"
Sep 17 19:02:08.236180 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:08.235992    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722" podUID="722d209e-1bc2-4607-87c4-ac89404a1548" containerName="main" containerID="cri-o://16356e7b7163f0d64ba1f9b2414859c1efcd0e7c6285b2a1ab7b2adcca159fbb" gracePeriod=60
Sep 17 19:02:08.316492 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:08.316459    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:08.736603 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:08.736560    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722" podUID="722d209e-1bc2-4607-87c4-ac89404a1548" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": dial tcp 10.134.0.89:8000: connect: connection refused"
Sep 17 19:02:08.920697 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:08.920657    2574 generic.go:356] "Generic (PLEG): container finished" podID="722d209e-1bc2-4607-87c4-ac89404a1548" containerID="16356e7b7163f0d64ba1f9b2414859c1efcd0e7c6285b2a1ab7b2adcca159fbb" exitCode=0
Sep 17 19:02:08.920862 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:08.920727    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722" event={"ID":"722d209e-1bc2-4607-87c4-ac89404a1548","Type":"ContainerDied","Data":"16356e7b7163f0d64ba1f9b2414859c1efcd0e7c6285b2a1ab7b2adcca159fbb"}
Sep 17 19:02:08.988256 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:08.988199    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:02:09.186017 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.185981    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-dshm\") pod \"722d209e-1bc2-4607-87c4-ac89404a1548\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") "
Sep 17 19:02:09.186180 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.186074    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-tmp-dir\") pod \"722d209e-1bc2-4607-87c4-ac89404a1548\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") "
Sep 17 19:02:09.186180 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.186111    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722d209e-1bc2-4607-87c4-ac89404a1548-tls-certs\") pod \"722d209e-1bc2-4607-87c4-ac89404a1548\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") "
Sep 17 19:02:09.186180 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.186131    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-model-cache\") pod \"722d209e-1bc2-4607-87c4-ac89404a1548\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") "
Sep 17 19:02:09.186180 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.186158    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-home\") pod \"722d209e-1bc2-4607-87c4-ac89404a1548\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") "
Sep 17 19:02:09.186417 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.186189    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sb4j\" (UniqueName: \"kubernetes.io/projected/722d209e-1bc2-4607-87c4-ac89404a1548-kube-api-access-5sb4j\") pod \"722d209e-1bc2-4607-87c4-ac89404a1548\" (UID: \"722d209e-1bc2-4607-87c4-ac89404a1548\") "
Sep 17 19:02:09.186417 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.186387    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "722d209e-1bc2-4607-87c4-ac89404a1548" (UID: "722d209e-1bc2-4607-87c4-ac89404a1548"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:09.186550 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.186416    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-model-cache" (OuterVolumeSpecName: "model-cache") pod "722d209e-1bc2-4607-87c4-ac89404a1548" (UID: "722d209e-1bc2-4607-87c4-ac89404a1548"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:09.186550 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.186470    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-home" (OuterVolumeSpecName: "home") pod "722d209e-1bc2-4607-87c4-ac89404a1548" (UID: "722d209e-1bc2-4607-87c4-ac89404a1548"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:09.188456 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.188426    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-dshm" (OuterVolumeSpecName: "dshm") pod "722d209e-1bc2-4607-87c4-ac89404a1548" (UID: "722d209e-1bc2-4607-87c4-ac89404a1548"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:09.188581 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.188484    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/722d209e-1bc2-4607-87c4-ac89404a1548-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "722d209e-1bc2-4607-87c4-ac89404a1548" (UID: "722d209e-1bc2-4607-87c4-ac89404a1548"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:09.188581 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.188561    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722d209e-1bc2-4607-87c4-ac89404a1548-kube-api-access-5sb4j" (OuterVolumeSpecName: "kube-api-access-5sb4j") pod "722d209e-1bc2-4607-87c4-ac89404a1548" (UID: "722d209e-1bc2-4607-87c4-ac89404a1548"). InnerVolumeSpecName "kube-api-access-5sb4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:02:09.286746 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.286721    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:02:09.286746 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.286745    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:02:09.287109 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.286758    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722d209e-1bc2-4607-87c4-ac89404a1548-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:02:09.287109 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.286770    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:02:09.287109 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.286782    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722d209e-1bc2-4607-87c4-ac89404a1548-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:02:09.287109 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.286793    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sb4j\" (UniqueName: \"kubernetes.io/projected/722d209e-1bc2-4607-87c4-ac89404a1548-kube-api-access-5sb4j\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:02:09.927632 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.927598    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"
Sep 17 19:02:09.927896 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.927592    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722" event={"ID":"722d209e-1bc2-4607-87c4-ac89404a1548","Type":"ContainerDied","Data":"42e7fd67e8394bd2da74d07603c2be4d538602e03914962356cbc3e4bcc0d3bd"}
Sep 17 19:02:09.927896 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.927760    2574 scope.go:117] "RemoveContainer" containerID="16356e7b7163f0d64ba1f9b2414859c1efcd0e7c6285b2a1ab7b2adcca159fbb"
Sep 17 19:02:09.951120 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.951096    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"]
Sep 17 19:02:09.954944 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:09.954919    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhp722"]
Sep 17 19:02:11.939044 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:11.939006    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="722d209e-1bc2-4607-87c4-ac89404a1548" path="/var/lib/kubelet/pods/722d209e-1bc2-4607-87c4-ac89404a1548/volumes"
Sep 17 19:02:12.215877 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:12.215801    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 19:02:12.235302 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:12.235279    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j2v2z_53270d0a-dc99-4f59-a2a3-2086917c92f2/ovn-acl-logging/0.log"
Sep 17 19:02:23.844507 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:23.844472    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"
Sep 17 19:02:25.529576 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.529538    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"]
Sep 17 19:02:25.530014 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.529879    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722d209e-1bc2-4607-87c4-ac89404a1548" containerName="main"
Sep 17 19:02:25.530014 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.529890    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="722d209e-1bc2-4607-87c4-ac89404a1548" containerName="main"
Sep 17 19:02:25.530014 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.529965    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="722d209e-1bc2-4607-87c4-ac89404a1548" containerName="main"
Sep 17 19:02:25.534833 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.534812    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.543810 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.543782    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"]
Sep 17 19:02:25.619157 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.619125    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.619157 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.619159    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.619356 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.619176    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/947bbff4-d11c-484e-b88f-3c0b2f81fbad-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.619356 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.619202    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.619356 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.619246    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7dhz\" (UniqueName: \"kubernetes.io/projected/947bbff4-d11c-484e-b88f-3c0b2f81fbad-kube-api-access-w7dhz\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.619356 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.619268    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.720486 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.720442    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.720700 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.720555    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.720700 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.720574    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.720700 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.720591    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/947bbff4-d11c-484e-b88f-3c0b2f81fbad-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.720700 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.720612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.720700 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.720650    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7dhz\" (UniqueName: \"kubernetes.io/projected/947bbff4-d11c-484e-b88f-3c0b2f81fbad-kube-api-access-w7dhz\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.720934 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.720857    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.720982 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.720950    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.721022 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.721004    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.723098 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.723067    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.723435 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.723418    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/947bbff4-d11c-484e-b88f-3c0b2f81fbad-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.729806 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.729781    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7dhz\" (UniqueName: \"kubernetes.io/projected/947bbff4-d11c-484e-b88f-3c0b2f81fbad-kube-api-access-w7dhz\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-v9hjp\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:25.845432 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:25.845341    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:26.118946 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:26.118918    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"]
Sep 17 19:02:26.120595 ip-10-0-132-0 kubenswrapper[2574]: W0917 19:02:26.120571    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947bbff4_d11c_484e_b88f_3c0b2f81fbad.slice/crio-99d44630291e36f38111b264ea8a7920a2a02014002e18ce98e861fc6b04c463 WatchSource:0}: Error finding container 99d44630291e36f38111b264ea8a7920a2a02014002e18ce98e861fc6b04c463: Status 404 returned error can't find the container with id 99d44630291e36f38111b264ea8a7920a2a02014002e18ce98e861fc6b04c463
Sep 17 19:02:27.021428 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:27.021393    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp" event={"ID":"947bbff4-d11c-484e-b88f-3c0b2f81fbad","Type":"ContainerStarted","Data":"1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb"}
Sep 17 19:02:27.021905 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:27.021450    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp" event={"ID":"947bbff4-d11c-484e-b88f-3c0b2f81fbad","Type":"ContainerStarted","Data":"99d44630291e36f38111b264ea8a7920a2a02014002e18ce98e861fc6b04c463"}
Sep 17 19:02:27.846066 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:27.846031    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:35.846491 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:35.846448    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:35.859238 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:35.859208    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:35.889592 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:35.889528    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp" podStartSLOduration=10.889497246 podStartE2EDuration="10.889497246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:02:27.04832919 +0000 UTC m=+3917.349151525" watchObservedRunningTime="2026-09-17 19:02:35.889497246 +0000 UTC m=+3926.190319623"
Sep 17 19:02:36.078790 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:36.078756    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:41.403377 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.403345    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:41.459599 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.459564    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"]
Sep 17 19:02:41.484910 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.484877    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"]
Sep 17 19:02:41.494129 ip-10-0-132-0 kubenswrapper[2574]: E0917 19:02:41.494085    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:02:41.494129 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:41.494129 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:41.494129 ip-10-0-132-0 kubenswrapper[2574]:  > containerID="1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb" cmd=["/bin/sleep","15"]
Sep 17 19:02:41.494467 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.494147    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp" message="" err=<
Sep 17 19:02:41.494467 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:41.494467 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:41.494467 ip-10-0-132-0 kubenswrapper[2574]:  >
Sep 17 19:02:41.494467 ip-10-0-132-0 kubenswrapper[2574]: E0917 19:02:41.494163    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:02:41.494467 ip-10-0-132-0 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:41.494467 ip-10-0-132-0 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:41.494467 ip-10-0-132-0 kubenswrapper[2574]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp" podUID="947bbff4-d11c-484e-b88f-3c0b2f81fbad" containerName="main" containerID="cri-o://1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb"
Sep 17 19:02:41.494467 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.494206    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp" podUID="947bbff4-d11c-484e-b88f-3c0b2f81fbad" containerName="main" containerID="cri-o://1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb" gracePeriod=60
Sep 17 19:02:41.843646 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.843618    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:41.963289 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.963199    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-home\") pod \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") "
Sep 17 19:02:41.963289 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.963245    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/947bbff4-d11c-484e-b88f-3c0b2f81fbad-tls-certs\") pod \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") "
Sep 17 19:02:41.963289 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.963267    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7dhz\" (UniqueName: \"kubernetes.io/projected/947bbff4-d11c-484e-b88f-3c0b2f81fbad-kube-api-access-w7dhz\") pod \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") "
Sep 17 19:02:41.963566 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.963433    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-dshm\") pod \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") "
Sep 17 19:02:41.963566 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.963502    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-tmp-dir\") pod \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") "
Sep 17 19:02:41.963566 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.963540    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-home" (OuterVolumeSpecName: "home") pod "947bbff4-d11c-484e-b88f-3c0b2f81fbad" (UID: "947bbff4-d11c-484e-b88f-3c0b2f81fbad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:41.963748 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.963573    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-model-cache\") pod \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\" (UID: \"947bbff4-d11c-484e-b88f-3c0b2f81fbad\") "
Sep 17 19:02:41.963810 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.963758    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "947bbff4-d11c-484e-b88f-3c0b2f81fbad" (UID: "947bbff4-d11c-484e-b88f-3c0b2f81fbad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:41.963866 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.963812    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-home\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:02:41.963866 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.963843    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-model-cache" (OuterVolumeSpecName: "model-cache") pod "947bbff4-d11c-484e-b88f-3c0b2f81fbad" (UID: "947bbff4-d11c-484e-b88f-3c0b2f81fbad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:41.965815 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.965792    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-dshm" (OuterVolumeSpecName: "dshm") pod "947bbff4-d11c-484e-b88f-3c0b2f81fbad" (UID: "947bbff4-d11c-484e-b88f-3c0b2f81fbad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:41.965905 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.965860    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947bbff4-d11c-484e-b88f-3c0b2f81fbad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "947bbff4-d11c-484e-b88f-3c0b2f81fbad" (UID: "947bbff4-d11c-484e-b88f-3c0b2f81fbad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:41.965905 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:41.965865    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947bbff4-d11c-484e-b88f-3c0b2f81fbad-kube-api-access-w7dhz" (OuterVolumeSpecName: "kube-api-access-w7dhz") pod "947bbff4-d11c-484e-b88f-3c0b2f81fbad" (UID: "947bbff4-d11c-484e-b88f-3c0b2f81fbad"). InnerVolumeSpecName "kube-api-access-w7dhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:02:42.064388 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.064345    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-tmp-dir\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:02:42.064388 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.064382    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-model-cache\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:02:42.064388 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.064392    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/947bbff4-d11c-484e-b88f-3c0b2f81fbad-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:02:42.064669 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.064400    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7dhz\" (UniqueName: \"kubernetes.io/projected/947bbff4-d11c-484e-b88f-3c0b2f81fbad-kube-api-access-w7dhz\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:02:42.064669 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.064410    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/947bbff4-d11c-484e-b88f-3c0b2f81fbad-dshm\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:02:42.100963 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.100926    2574 generic.go:356] "Generic (PLEG): container finished" podID="947bbff4-d11c-484e-b88f-3c0b2f81fbad" containerID="1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb" exitCode=0
Sep 17 19:02:42.101129 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.100982    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"
Sep 17 19:02:42.101129 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.101004    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp" event={"ID":"947bbff4-d11c-484e-b88f-3c0b2f81fbad","Type":"ContainerDied","Data":"1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb"}
Sep 17 19:02:42.101129 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.101046    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp" event={"ID":"947bbff4-d11c-484e-b88f-3c0b2f81fbad","Type":"ContainerDied","Data":"99d44630291e36f38111b264ea8a7920a2a02014002e18ce98e861fc6b04c463"}
Sep 17 19:02:42.101129 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.101067    2574 scope.go:117] "RemoveContainer" containerID="1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb"
Sep 17 19:02:42.111064 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.111050    2574 scope.go:117] "RemoveContainer" containerID="1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb"
Sep 17 19:02:42.111321 ip-10-0-132-0 kubenswrapper[2574]: E0917 19:02:42.111306    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb\": container with ID starting with 1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb not found: ID does not exist" containerID="1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb"
Sep 17 19:02:42.111361 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.111329    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb"} err="failed to get container status \"1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb\": rpc error: code = NotFound desc = could not find container \"1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb\": container with ID starting with 1b093955f1dce1045523bd0551af5d7079ab17b7a72ab3c81a6dbd516a6fb7cb not found: ID does not exist"
Sep 17 19:02:42.130965 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.130938    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"]
Sep 17 19:02:42.132989 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.132971    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-v9hjp"]
Sep 17 19:02:42.939780 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:42.939749    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:43.844926 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:43.844884    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl" podUID="8c88884f-712e-4dcf-9aba-833df3d6259b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:02:43.938678 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:43.938640    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="947bbff4-d11c-484e-b88f-3c0b2f81fbad" path="/var/lib/kubelet/pods/947bbff4-d11c-484e-b88f-3c0b2f81fbad/volumes"
Sep 17 19:02:53.845040 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:53.844950    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl" podUID="8c88884f-712e-4dcf-9aba-833df3d6259b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:02:56.479503 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:56.479456    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl" podUID="8c88884f-712e-4dcf-9aba-833df3d6259b" containerName="main" containerID="cri-o://3031a49fc81305e2213e6c5dc8cf8b894770c69f13f0a3a25a1c453586742cac" gracePeriod=45
Sep 17 19:02:57.514618 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:57.514584    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:57.816328 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:02:57.816255    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:03.844909 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:03.844856    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl" podUID="8c88884f-712e-4dcf-9aba-833df3d6259b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:03:03.845604 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:03.844970    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"
Sep 17 19:03:04.841018 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:04.840986    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:07.865562 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:07.865503    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:08.163792 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:08.163683    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:11.263924 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:11.263882    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:11.364818 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:11.364776    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:13.845023 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:13.844985    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl" podUID="8c88884f-712e-4dcf-9aba-833df3d6259b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:03:23.845211 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:23.845170    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl" podUID="8c88884f-712e-4dcf-9aba-833df3d6259b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:03:23.978221 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:23.978185    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:26.741715 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:26.741653    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"
Sep 17 19:03:26.829377 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:26.829343    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkfps\" (UniqueName: \"kubernetes.io/projected/8c88884f-712e-4dcf-9aba-833df3d6259b-kube-api-access-gkfps\") pod \"8c88884f-712e-4dcf-9aba-833df3d6259b\" (UID: \"8c88884f-712e-4dcf-9aba-833df3d6259b\") "
Sep 17 19:03:26.829580 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:26.829421    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c88884f-712e-4dcf-9aba-833df3d6259b-tls-certs\") pod \"8c88884f-712e-4dcf-9aba-833df3d6259b\" (UID: \"8c88884f-712e-4dcf-9aba-833df3d6259b\") "
Sep 17 19:03:26.831768 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:26.831734    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c88884f-712e-4dcf-9aba-833df3d6259b-kube-api-access-gkfps" (OuterVolumeSpecName: "kube-api-access-gkfps") pod "8c88884f-712e-4dcf-9aba-833df3d6259b" (UID: "8c88884f-712e-4dcf-9aba-833df3d6259b"). InnerVolumeSpecName "kube-api-access-gkfps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:03:26.831884 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:26.831806    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c88884f-712e-4dcf-9aba-833df3d6259b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c88884f-712e-4dcf-9aba-833df3d6259b" (UID: "8c88884f-712e-4dcf-9aba-833df3d6259b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:03:26.930051 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:26.930012    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c88884f-712e-4dcf-9aba-833df3d6259b-tls-certs\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:03:26.930051 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:26.930045    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkfps\" (UniqueName: \"kubernetes.io/projected/8c88884f-712e-4dcf-9aba-833df3d6259b-kube-api-access-gkfps\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:03:27.350305 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:27.350257    2574 generic.go:356] "Generic (PLEG): container finished" podID="8c88884f-712e-4dcf-9aba-833df3d6259b" containerID="3031a49fc81305e2213e6c5dc8cf8b894770c69f13f0a3a25a1c453586742cac" exitCode=0
Sep 17 19:03:27.350479 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:27.350314    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"
Sep 17 19:03:27.350479 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:27.350350    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl" event={"ID":"8c88884f-712e-4dcf-9aba-833df3d6259b","Type":"ContainerDied","Data":"3031a49fc81305e2213e6c5dc8cf8b894770c69f13f0a3a25a1c453586742cac"}
Sep 17 19:03:27.350479 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:27.350408    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl" event={"ID":"8c88884f-712e-4dcf-9aba-833df3d6259b","Type":"ContainerDied","Data":"35b5714b9d7678f659e30fcbdc94241f03958b1e8df5187e0d2a257fc3868ad8"}
Sep 17 19:03:27.350479 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:27.350432    2574 scope.go:117] "RemoveContainer" containerID="3031a49fc81305e2213e6c5dc8cf8b894770c69f13f0a3a25a1c453586742cac"
Sep 17 19:03:27.362001 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:27.361978    2574 scope.go:117] "RemoveContainer" containerID="3031a49fc81305e2213e6c5dc8cf8b894770c69f13f0a3a25a1c453586742cac"
Sep 17 19:03:27.362452 ip-10-0-132-0 kubenswrapper[2574]: E0917 19:03:27.362430    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3031a49fc81305e2213e6c5dc8cf8b894770c69f13f0a3a25a1c453586742cac\": container with ID starting with 3031a49fc81305e2213e6c5dc8cf8b894770c69f13f0a3a25a1c453586742cac not found: ID does not exist" containerID="3031a49fc81305e2213e6c5dc8cf8b894770c69f13f0a3a25a1c453586742cac"
Sep 17 19:03:27.362567 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:27.362459    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3031a49fc81305e2213e6c5dc8cf8b894770c69f13f0a3a25a1c453586742cac"} err="failed to get container status \"3031a49fc81305e2213e6c5dc8cf8b894770c69f13f0a3a25a1c453586742cac\": rpc error: code = NotFound desc = could not find container \"3031a49fc81305e2213e6c5dc8cf8b894770c69f13f0a3a25a1c453586742cac\": container with ID starting with 3031a49fc81305e2213e6c5dc8cf8b894770c69f13f0a3a25a1c453586742cac not found: ID does not exist"
Sep 17 19:03:27.387063 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:27.387038    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"]
Sep 17 19:03:27.390067 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:27.390046    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-8z6hl"]
Sep 17 19:03:27.848287 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:27.848255    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:27.938781 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:27.938748    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c88884f-712e-4dcf-9aba-833df3d6259b" path="/var/lib/kubelet/pods/8c88884f-712e-4dcf-9aba-833df3d6259b/volumes"
Sep 17 19:03:34.181074 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:34.181037    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:35.046872 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:35.046843    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:43.248495 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:43.248450    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:43.464565 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:43.464532    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:43.938792 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:43.938759    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:44.060362 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:44.060320    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:51.645801 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:51.645771    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:51.765639 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:03:51.765603    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:32.824413 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:32.824382    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:36.287286 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.287249    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bm5lk/must-gather-b6hwv"]
Sep 17 19:04:36.287687 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.287598    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="947bbff4-d11c-484e-b88f-3c0b2f81fbad" containerName="main"
Sep 17 19:04:36.287687 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.287610    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="947bbff4-d11c-484e-b88f-3c0b2f81fbad" containerName="main"
Sep 17 19:04:36.287687 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.287629    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c88884f-712e-4dcf-9aba-833df3d6259b" containerName="main"
Sep 17 19:04:36.287687 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.287635    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c88884f-712e-4dcf-9aba-833df3d6259b" containerName="main"
Sep 17 19:04:36.287847 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.287695    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="947bbff4-d11c-484e-b88f-3c0b2f81fbad" containerName="main"
Sep 17 19:04:36.287847 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.287704    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c88884f-712e-4dcf-9aba-833df3d6259b" containerName="main"
Sep 17 19:04:36.291870 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.291841    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bm5lk/must-gather-b6hwv"
Sep 17 19:04:36.295121 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.295102    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bm5lk\"/\"default-dockercfg-j9tm7\""
Sep 17 19:04:36.295221 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.295132    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bm5lk\"/\"openshift-service-ca.crt\""
Sep 17 19:04:36.296069 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.296021    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bm5lk\"/\"kube-root-ca.crt\""
Sep 17 19:04:36.306762 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.306739    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bm5lk/must-gather-b6hwv"]
Sep 17 19:04:36.319653 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.319628    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:36.397312 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.397283    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kp8n\" (UniqueName: \"kubernetes.io/projected/8161f862-08ea-44f4-9928-609da6dd7ffc-kube-api-access-7kp8n\") pod \"must-gather-b6hwv\" (UID: \"8161f862-08ea-44f4-9928-609da6dd7ffc\") " pod="openshift-must-gather-bm5lk/must-gather-b6hwv"
Sep 17 19:04:36.397445 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.397395    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8161f862-08ea-44f4-9928-609da6dd7ffc-must-gather-output\") pod \"must-gather-b6hwv\" (UID: \"8161f862-08ea-44f4-9928-609da6dd7ffc\") " pod="openshift-must-gather-bm5lk/must-gather-b6hwv"
Sep 17 19:04:36.416282 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.416257    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:36.498417 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.498386    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8161f862-08ea-44f4-9928-609da6dd7ffc-must-gather-output\") pod \"must-gather-b6hwv\" (UID: \"8161f862-08ea-44f4-9928-609da6dd7ffc\") " pod="openshift-must-gather-bm5lk/must-gather-b6hwv"
Sep 17 19:04:36.498578 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.498425    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kp8n\" (UniqueName: \"kubernetes.io/projected/8161f862-08ea-44f4-9928-609da6dd7ffc-kube-api-access-7kp8n\") pod \"must-gather-b6hwv\" (UID: \"8161f862-08ea-44f4-9928-609da6dd7ffc\") " pod="openshift-must-gather-bm5lk/must-gather-b6hwv"
Sep 17 19:04:36.498816 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.498796    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8161f862-08ea-44f4-9928-609da6dd7ffc-must-gather-output\") pod \"must-gather-b6hwv\" (UID: \"8161f862-08ea-44f4-9928-609da6dd7ffc\") " pod="openshift-must-gather-bm5lk/must-gather-b6hwv"
Sep 17 19:04:36.507629 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.507605    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kp8n\" (UniqueName: \"kubernetes.io/projected/8161f862-08ea-44f4-9928-609da6dd7ffc-kube-api-access-7kp8n\") pod \"must-gather-b6hwv\" (UID: \"8161f862-08ea-44f4-9928-609da6dd7ffc\") " pod="openshift-must-gather-bm5lk/must-gather-b6hwv"
Sep 17 19:04:36.601383 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.601315    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bm5lk/must-gather-b6hwv"
Sep 17 19:04:36.877392 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.877354    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bm5lk/must-gather-b6hwv"]
Sep 17 19:04:36.878820 ip-10-0-132-0 kubenswrapper[2574]: W0917 19:04:36.878775    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8161f862_08ea_44f4_9928_609da6dd7ffc.slice/crio-e8c9fa5014aa3014ba5e10211349ea5387c07801e91c277872f6765df152ab0c WatchSource:0}: Error finding container e8c9fa5014aa3014ba5e10211349ea5387c07801e91c277872f6765df152ab0c: Status 404 returned error can't find the container with id e8c9fa5014aa3014ba5e10211349ea5387c07801e91c277872f6765df152ab0c
Sep 17 19:04:36.880626 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:36.880609    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:04:37.727597 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:37.727507    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bm5lk/must-gather-b6hwv" event={"ID":"8161f862-08ea-44f4-9928-609da6dd7ffc","Type":"ContainerStarted","Data":"e8c9fa5014aa3014ba5e10211349ea5387c07801e91c277872f6765df152ab0c"}
Sep 17 19:04:40.144183 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:40.144148    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:43.768267 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:43.768219    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bm5lk/must-gather-b6hwv" event={"ID":"8161f862-08ea-44f4-9928-609da6dd7ffc","Type":"ContainerStarted","Data":"b61d97ca4d6f76629811e0314da52a58670a4567667501fca0a9ce286197104a"}
Sep 17 19:04:43.768737 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:43.768280    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bm5lk/must-gather-b6hwv" event={"ID":"8161f862-08ea-44f4-9928-609da6dd7ffc","Type":"ContainerStarted","Data":"d53593e9811c6695c7a145a7f7093e5b5f578aca6900cdcd39398adf05432e0c"}
Sep 17 19:04:43.788108 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:43.788060    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bm5lk/must-gather-b6hwv" podStartSLOduration=1.420239195 podStartE2EDuration="7.788046069s" totalImagesPullingTime="6.367806874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:04:43.78658041 +0000 UTC m=+4054.087402744" watchObservedRunningTime="2026-09-17 19:04:43.788046069 +0000 UTC m=+4054.088868411"
Sep 17 19:04:46.574657 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:46.574618    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:46.688589 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:04:46.688547    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:03.496989 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:03.496935    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-5sq9k_bb91b6c1-b60e-40d8-a953-a0c3bc438381/manager/0.log"
Sep 17 19:05:03.772779 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:03.772754    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-6g482_1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da/limitador/0.log"
Sep 17 19:05:03.833802 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:03.833752    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8j28v_288ad38b-180c-4a5c-83bd-ad3b309d3c62/manager/0.log"
Sep 17 19:05:06.911085 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:06.911050    2574 generic.go:356] "Generic (PLEG): container finished" podID="8161f862-08ea-44f4-9928-609da6dd7ffc" containerID="d53593e9811c6695c7a145a7f7093e5b5f578aca6900cdcd39398adf05432e0c" exitCode=0
Sep 17 19:05:06.911496 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:06.911124    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bm5lk/must-gather-b6hwv" event={"ID":"8161f862-08ea-44f4-9928-609da6dd7ffc","Type":"ContainerDied","Data":"d53593e9811c6695c7a145a7f7093e5b5f578aca6900cdcd39398adf05432e0c"}
Sep 17 19:05:06.911496 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:06.911475    2574 scope.go:117] "RemoveContainer" containerID="d53593e9811c6695c7a145a7f7093e5b5f578aca6900cdcd39398adf05432e0c"
Sep 17 19:05:07.354856 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:07.354823    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bm5lk_must-gather-b6hwv_8161f862-08ea-44f4-9928-609da6dd7ffc/gather/0.log"
Sep 17 19:05:07.953639 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:07.953530    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:08.216587 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:08.216480    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:08.389556 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:08.389506    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:11.914494 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:11.914460    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4kg4f_c6126893-24bd-4045-a954-7dd893aff7e8/global-pull-secret-syncer/0.log"
Sep 17 19:05:12.327832 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:12.327800    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log"
Sep 17 19:05:12.761946 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:12.761917    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-0.ec2.internal_ef7d3c93d84d7fbd4bfd4bfbc5d80220/haproxy/0.log"
Sep 17 19:05:12.962456 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:12.962416    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-bm5lk/must-gather-b6hwv"]
Sep 17 19:05:12.962943 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:12.962621    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-bm5lk/must-gather-b6hwv" podUID="8161f862-08ea-44f4-9928-609da6dd7ffc" containerName="copy" containerID="cri-o://b61d97ca4d6f76629811e0314da52a58670a4567667501fca0a9ce286197104a" gracePeriod=2
Sep 17 19:05:12.972021 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:12.971989    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-bm5lk/must-gather-b6hwv"]
Sep 17 19:05:13.200840 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.200815    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bm5lk_must-gather-b6hwv_8161f862-08ea-44f4-9928-609da6dd7ffc/copy/0.log"
Sep 17 19:05:13.202822 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.202804    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bm5lk/must-gather-b6hwv"
Sep 17 19:05:13.209959 ip-10-0-132-0 kubenswrapper[2574]: E0917 19:05:13.209934    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-b6hwv\" is forbidden: User \"system:node:ip-10-0-132-0.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bm5lk\": no relationship found between node 'ip-10-0-132-0.ec2.internal' and this object" podUID="8161f862-08ea-44f4-9928-609da6dd7ffc" pod="openshift-must-gather-bm5lk/must-gather-b6hwv"
Sep 17 19:05:13.322996 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.322953    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kp8n\" (UniqueName: \"kubernetes.io/projected/8161f862-08ea-44f4-9928-609da6dd7ffc-kube-api-access-7kp8n\") pod \"8161f862-08ea-44f4-9928-609da6dd7ffc\" (UID: \"8161f862-08ea-44f4-9928-609da6dd7ffc\") "
Sep 17 19:05:13.323168 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.323129    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8161f862-08ea-44f4-9928-609da6dd7ffc-must-gather-output\") pod \"8161f862-08ea-44f4-9928-609da6dd7ffc\" (UID: \"8161f862-08ea-44f4-9928-609da6dd7ffc\") "
Sep 17 19:05:13.325617 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.325546    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8161f862-08ea-44f4-9928-609da6dd7ffc-kube-api-access-7kp8n" (OuterVolumeSpecName: "kube-api-access-7kp8n") pod "8161f862-08ea-44f4-9928-609da6dd7ffc" (UID: "8161f862-08ea-44f4-9928-609da6dd7ffc"). InnerVolumeSpecName "kube-api-access-7kp8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:05:13.331497 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.331467    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8161f862-08ea-44f4-9928-609da6dd7ffc-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "8161f862-08ea-44f4-9928-609da6dd7ffc" (UID: "8161f862-08ea-44f4-9928-609da6dd7ffc"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:05:13.424108 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.424065    2574 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8161f862-08ea-44f4-9928-609da6dd7ffc-must-gather-output\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:05:13.424108 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.424101    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kp8n\" (UniqueName: \"kubernetes.io/projected/8161f862-08ea-44f4-9928-609da6dd7ffc-kube-api-access-7kp8n\") on node \"ip-10-0-132-0.ec2.internal\" DevicePath \"\""
Sep 17 19:05:13.937955 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.937921    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8161f862-08ea-44f4-9928-609da6dd7ffc" path="/var/lib/kubelet/pods/8161f862-08ea-44f4-9928-609da6dd7ffc/volumes"
Sep 17 19:05:13.949474 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.949454    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bm5lk_must-gather-b6hwv_8161f862-08ea-44f4-9928-609da6dd7ffc/copy/0.log"
Sep 17 19:05:13.949807 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.949782    2574 generic.go:356] "Generic (PLEG): container finished" podID="8161f862-08ea-44f4-9928-609da6dd7ffc" containerID="b61d97ca4d6f76629811e0314da52a58670a4567667501fca0a9ce286197104a" exitCode=143
Sep 17 19:05:13.949889 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.949834    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bm5lk/must-gather-b6hwv"
Sep 17 19:05:13.949945 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.949889    2574 scope.go:117] "RemoveContainer" containerID="b61d97ca4d6f76629811e0314da52a58670a4567667501fca0a9ce286197104a"
Sep 17 19:05:13.958789 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.958767    2574 scope.go:117] "RemoveContainer" containerID="d53593e9811c6695c7a145a7f7093e5b5f578aca6900cdcd39398adf05432e0c"
Sep 17 19:05:13.974988 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.974860    2574 scope.go:117] "RemoveContainer" containerID="b61d97ca4d6f76629811e0314da52a58670a4567667501fca0a9ce286197104a"
Sep 17 19:05:13.975190 ip-10-0-132-0 kubenswrapper[2574]: E0917 19:05:13.975107    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61d97ca4d6f76629811e0314da52a58670a4567667501fca0a9ce286197104a\": container with ID starting with b61d97ca4d6f76629811e0314da52a58670a4567667501fca0a9ce286197104a not found: ID does not exist" containerID="b61d97ca4d6f76629811e0314da52a58670a4567667501fca0a9ce286197104a"
Sep 17 19:05:13.975190 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.975140    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61d97ca4d6f76629811e0314da52a58670a4567667501fca0a9ce286197104a"} err="failed to get container status \"b61d97ca4d6f76629811e0314da52a58670a4567667501fca0a9ce286197104a\": rpc error: code = NotFound desc = could not find container \"b61d97ca4d6f76629811e0314da52a58670a4567667501fca0a9ce286197104a\": container with ID starting with b61d97ca4d6f76629811e0314da52a58670a4567667501fca0a9ce286197104a not found: ID does not exist"
Sep 17 19:05:13.975190 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.975162    2574 scope.go:117] "RemoveContainer" containerID="d53593e9811c6695c7a145a7f7093e5b5f578aca6900cdcd39398adf05432e0c"
Sep 17 19:05:13.975452 ip-10-0-132-0 kubenswrapper[2574]: E0917 19:05:13.975433    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53593e9811c6695c7a145a7f7093e5b5f578aca6900cdcd39398adf05432e0c\": container with ID starting with d53593e9811c6695c7a145a7f7093e5b5f578aca6900cdcd39398adf05432e0c not found: ID does not exist" containerID="d53593e9811c6695c7a145a7f7093e5b5f578aca6900cdcd39398adf05432e0c"
Sep 17 19:05:13.975493 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:13.975466    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53593e9811c6695c7a145a7f7093e5b5f578aca6900cdcd39398adf05432e0c"} err="failed to get container status \"d53593e9811c6695c7a145a7f7093e5b5f578aca6900cdcd39398adf05432e0c\": rpc error: code = NotFound desc = could not find container \"d53593e9811c6695c7a145a7f7093e5b5f578aca6900cdcd39398adf05432e0c\": container with ID starting with d53593e9811c6695c7a145a7f7093e5b5f578aca6900cdcd39398adf05432e0c not found: ID does not exist"
Sep 17 19:05:16.873925 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:16.873898    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-5sq9k_bb91b6c1-b60e-40d8-a953-a0c3bc438381/manager/0.log"
Sep 17 19:05:18.712196 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:18.712169    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-6g482_1b57ec3f-030a-47e0-9bf3-ac3e1fafe0da/limitador/0.log"
Sep 17 19:05:18.815364 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:18.815329    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8j28v_288ad38b-180c-4a5c-83bd-ad3b309d3c62/manager/0.log"
Sep 17 19:05:19.407257 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:19.407218    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:19.623723 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:19.623689    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6jhpj_229dfe1c-397d-48b4-b1d4-e3c5e6429339/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:21.343400 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.343365    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"]
Sep 17 19:05:21.343829 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.343745    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8161f862-08ea-44f4-9928-609da6dd7ffc" containerName="copy"
Sep 17 19:05:21.343829 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.343756    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8161f862-08ea-44f4-9928-609da6dd7ffc" containerName="copy"
Sep 17 19:05:21.343829 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.343766    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8161f862-08ea-44f4-9928-609da6dd7ffc" containerName="gather"
Sep 17 19:05:21.343829 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.343771    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8161f862-08ea-44f4-9928-609da6dd7ffc" containerName="gather"
Sep 17 19:05:21.343829 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.343825    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8161f862-08ea-44f4-9928-609da6dd7ffc" containerName="gather"
Sep 17 19:05:21.344021 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.343837    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8161f862-08ea-44f4-9928-609da6dd7ffc" containerName="copy"
Sep 17 19:05:21.345786 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.345769    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.348580 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.348559    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f4j9s\"/\"openshift-service-ca.crt\""
Sep 17 19:05:21.348832 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.348809    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f4j9s\"/\"default-dockercfg-9g8bv\""
Sep 17 19:05:21.349535 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.349508    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f4j9s\"/\"kube-root-ca.crt\""
Sep 17 19:05:21.380032 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.380004    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"]
Sep 17 19:05:21.493405 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.493379    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a851748d-233c-492d-9e1f-7fb1d98f7b79-sys\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.493546 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.493418    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a851748d-233c-492d-9e1f-7fb1d98f7b79-podres\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.493546 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.493444    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a851748d-233c-492d-9e1f-7fb1d98f7b79-proc\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.493546 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.493460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a851748d-233c-492d-9e1f-7fb1d98f7b79-lib-modules\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.493546 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.493501    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npfrj\" (UniqueName: \"kubernetes.io/projected/a851748d-233c-492d-9e1f-7fb1d98f7b79-kube-api-access-npfrj\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.518785 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.518754    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-sm8vh_12abcf49-2ce7-4fa4-ac1a-b8f6e3955e2a/monitoring-plugin/0.log"
Sep 17 19:05:21.594382 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.594282    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npfrj\" (UniqueName: \"kubernetes.io/projected/a851748d-233c-492d-9e1f-7fb1d98f7b79-kube-api-access-npfrj\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.594382 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.594341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a851748d-233c-492d-9e1f-7fb1d98f7b79-sys\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.594382 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.594366    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a851748d-233c-492d-9e1f-7fb1d98f7b79-podres\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.594704 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.594390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a851748d-233c-492d-9e1f-7fb1d98f7b79-proc\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.594704 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.594404    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a851748d-233c-492d-9e1f-7fb1d98f7b79-lib-modules\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.594704 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.594498    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a851748d-233c-492d-9e1f-7fb1d98f7b79-sys\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.594704 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.594534    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a851748d-233c-492d-9e1f-7fb1d98f7b79-proc\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.594704 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.594544    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a851748d-233c-492d-9e1f-7fb1d98f7b79-lib-modules\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.594704 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.594542    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a851748d-233c-492d-9e1f-7fb1d98f7b79-podres\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.610934 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.610900    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npfrj\" (UniqueName: \"kubernetes.io/projected/a851748d-233c-492d-9e1f-7fb1d98f7b79-kube-api-access-npfrj\") pod \"perf-node-gather-daemonset-t8kt4\" (UID: \"a851748d-233c-492d-9e1f-7fb1d98f7b79\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.655681 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.655643    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:21.954352 ip-10-0-132-0 kubenswrapper[2574]: W0917 19:05:21.954312    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda851748d_233c_492d_9e1f_7fb1d98f7b79.slice/crio-37d4baf8cf97814b7721057c77c29cbfeeb794dc02127a73b486b2f3f91168db WatchSource:0}: Error finding container 37d4baf8cf97814b7721057c77c29cbfeeb794dc02127a73b486b2f3f91168db: Status 404 returned error can't find the container with id 37d4baf8cf97814b7721057c77c29cbfeeb794dc02127a73b486b2f3f91168db
Sep 17 19:05:21.955224 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.955199    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"]
Sep 17 19:05:21.999199 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:21.999174    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4" event={"ID":"a851748d-233c-492d-9e1f-7fb1d98f7b79","Type":"ContainerStarted","Data":"37d4baf8cf97814b7721057c77c29cbfeeb794dc02127a73b486b2f3f91168db"}
Sep 17 19:05:22.122948 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:22.122912    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpx8b_0cecc0ad-f2e1-488b-a2f7-d46f669e8012/node-exporter/0.log"
Sep 17 19:05:22.181434 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:22.181403    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpx8b_0cecc0ad-f2e1-488b-a2f7-d46f669e8012/kube-rbac-proxy/0.log"
Sep 17 19:05:22.257955 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:22.257932    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpx8b_0cecc0ad-f2e1-488b-a2f7-d46f669e8012/init-textfile/0.log"
Sep 17 19:05:23.006268 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:23.006228    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4" event={"ID":"a851748d-233c-492d-9e1f-7fb1d98f7b79","Type":"ContainerStarted","Data":"dd915df6bb429335b5885e2b7d10446c83352071eef88ad0ce97741e638ef531"}
Sep 17 19:05:23.006696 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:23.006352    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:23.040637 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:23.040588    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4" podStartSLOduration=2.040571157 podStartE2EDuration="2.040571157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:05:23.039985578 +0000 UTC m=+4093.340807901" watchObservedRunningTime="2026-09-17 19:05:23.040571157 +0000 UTC m=+4093.341393480"
Sep 17 19:05:23.221083 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:23.221050    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-8l6qg_043673ea-e295-4b8a-a582-3af684b752b1/prometheus-operator-admission-webhook/0.log"
Sep 17 19:05:24.996222 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:24.996190    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-gp5gh_15845110-5c22-4b7c-b600-86ec6e146fe1/networking-console-plugin/0.log"
Sep 17 19:05:26.373857 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:26.373826    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-cb4977c64-r99sq_23b7236a-7fd4-44d2-bdcd-0faa73cf4e28/console/0.log"
Sep 17 19:05:26.528117 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:26.528086    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-8qvts_d79f9b27-81c4-47aa-b378-ed1d73cd2b03/download-server/0.log"
Sep 17 19:05:27.494105 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:27.494074    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-tqbd2_39c9f091-33e4-4108-beec-d30e38305a30/volume-data-source-validator/0.log"
Sep 17 19:05:29.021715 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:29.021686    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-t8kt4"
Sep 17 19:05:29.232683 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:29.232653    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pbkvq_da415339-7f86-4698-94a7-625f0e5015b2/dns/0.log"
Sep 17 19:05:29.296470 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:29.296398    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pbkvq_da415339-7f86-4698-94a7-625f0e5015b2/kube-rbac-proxy/0.log"
Sep 17 19:05:29.449714 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:29.449687    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4mlb8_e2c380ad-08a8-4689-b88f-f07b659e6314/dns-node-resolver/0.log"
Sep 17 19:05:30.455552 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:30.455451    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-795f647d56-d6k2l_eb0a4576-e4c8-423e-b504-1916fb761cdc/registry/0.log"
Sep 17 19:05:30.539805 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:30.539765    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4vwlx_a8fb6002-f122-4cc8-a73b-3d1e45addd6e/node-ca/0.log"
Sep 17 19:05:33.905060 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:33.904963    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jl2b8_bf9e50fc-d402-4599-8278-cc186d5a07ab/serve-healthcheck-canary/0.log"
Sep 17 19:05:34.924437 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:34.924403    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-flfs2_04d65cc5-2f68-45e9-8359-fff61bd12996/insights-operator/0.log"
Sep 17 19:05:34.925825 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:34.925806    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-flfs2_04d65cc5-2f68-45e9-8359-fff61bd12996/insights-operator/1.log"
Sep 17 19:05:35.302282 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:35.302248    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-55r82_4b53aee2-867a-44e4-91fa-f894cde05535/kube-rbac-proxy/0.log"
Sep 17 19:05:35.382599 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:35.382573    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-55r82_4b53aee2-867a-44e4-91fa-f894cde05535/exporter/0.log"
Sep 17 19:05:35.450744 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:35.450718    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-55r82_4b53aee2-867a-44e4-91fa-f894cde05535/extractor/0.log"
Sep 17 19:05:39.105205 ip-10-0-132-0 kubenswrapper[2574]: I0917 19:05:39.105179    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jvpzp_eb0cac6e-e23d-43da-a3c0-e351524c0262/openshift-lws-operator/0.log"