Jul 08 06:52:58.106919 ip-10-0-140-110 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 08 06:52:58.106934 ip-10-0-140-110 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 08 06:52:58.106944 ip-10-0-140-110 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 08 06:52:58.107371 ip-10-0-140-110 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 08 06:53:08.125330 ip-10-0-140-110 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 08 06:53:08.125350 ip-10-0-140-110 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1bfb79930d044229b87cdd3b280f4240 --
Jul 08 06:54:44.082396 ip-10-0-140-110 systemd[1]: Starting Kubernetes Kubelet...
Jul 08 06:54:44.591608 ip-10-0-140-110 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 06:54:44.591608 ip-10-0-140-110 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 08 06:54:44.591608 ip-10-0-140-110 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 06:54:44.591608 ip-10-0-140-110 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 08 06:54:44.591608 ip-10-0-140-110 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 06:54:44.591608 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.511556    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 08 06:54:44.591608 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517183    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 06:54:44.591608 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517194    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 06:54:44.591608 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517199    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 06:54:44.591608 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517202    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 06:54:44.591608 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517205    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 06:54:44.591608 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517208    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 06:54:44.591608 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517211    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 06:54:44.591608 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517214    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517217    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517219    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517222    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517225    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517227    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517230    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517232    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517235    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517237    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517240    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517242    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517245    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517248    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517251    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517253    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517257    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517259    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517265    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517268    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 06:54:44.804474 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517271    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 06:54:44.590960 ip-10-0-140-110 systemd[1]: Started Kubernetes Kubelet.
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517274    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517276    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517279    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517282    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517284    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517287    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517289    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517292    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517294    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517297    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517299    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517302    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517304    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517307    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517310    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517312    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517316    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517318    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517321    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 06:54:44.819968 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517329    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517332    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517335    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517340    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517344    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517347    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517350    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517353    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517355    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517359    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517361    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517364    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517367    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517370    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517372    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517375    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517377    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517380    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517382    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 06:54:44.822814 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517385    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517387    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517390    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517393    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517395    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517399    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517401    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517404    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517406    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517409    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517412    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517414    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517417    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517420    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517423    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517425    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517428    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517430    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517433    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517436    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 06:54:44.823975 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517438    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517441    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517443    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517446    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517450    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517453    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517456    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517458    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517461    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517463    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517468    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517471    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.517474    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518133    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518139    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518142    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518145    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518148    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518166    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 06:54:44.825118 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518168    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518171    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518174    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518177    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518180    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518183    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518185    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518188    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518191    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518196    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518200    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518203    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518206    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518208    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518211    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518215    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518218    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518221    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518224    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 06:54:44.826281 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518226    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518229    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518231    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518234    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518236    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518239    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518242    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518244    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518247    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518249    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518252    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518255    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518258    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518260    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518262    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518265    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518268    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518270    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518272    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 06:54:44.827026 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518277    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518280    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518282    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518285    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518288    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518291    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518293    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518296    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518299    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518301    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518303    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518306    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518308    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518311    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518314    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518316    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518318    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518321    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518324    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518327    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 06:54:44.827866 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518329    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518332    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518334    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518336    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518339    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518341    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518344    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518346    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518349    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518352    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518354    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518357    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518362    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518366    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518368    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518371    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518374    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518377    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518381    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 06:54:44.828493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518384    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518387    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518390    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518392    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518395    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518397    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518400    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518402    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518405    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518407    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518410    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518412    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518414    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518417    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518420    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518423    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518784    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518789    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518792    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518795    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 06:54:44.829119 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518798    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518801    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518803    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518806    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518808    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518814    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518818    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518821    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518824    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518827    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518830    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518833    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518835    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518838    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518840    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518843    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518846    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518849    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518851    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 06:54:44.829744 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518854    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518856    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518859    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518861    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518864    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518866    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518870    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518874    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518876    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518879    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518881    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518884    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518886    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518889    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518891    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518894    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518897    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518901    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518905    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 06:54:44.830379 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518908    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518911    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518914    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518918    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518921    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518924    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518926    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518928    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518931    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518934    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518936    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518939    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518941    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518944    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518946    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518949    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518951    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518954    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518956    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518958    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 06:54:44.831005 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518961    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518963    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518966    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518968    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518971    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518973    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518976    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518978    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518982    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518984    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518987    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518991    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518993    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518996    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.518998    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519005    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519007    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519011    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519014    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519016    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 06:54:44.831640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519019    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519021    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519024    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519026    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519029    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519031    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519034    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519036    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519039    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519041    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519054    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519058    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519060    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519063    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519065    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519068    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519070    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519140    2582 flags.go:64] FLAG: --address="0.0.0.0"
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519147    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519172    2582 flags.go:64] FLAG: --anonymous-auth="true"
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519177    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 08 06:54:44.832472 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519182    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519186    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519191    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519195    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519198    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519201    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519206    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519209    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519212    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519216    2582 flags.go:64] FLAG: --cgroup-root=""
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519218    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519221    2582 flags.go:64] FLAG: --client-ca-file=""
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519224    2582 flags.go:64] FLAG: --cloud-provider="external"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519228    2582 flags.go:64] FLAG: --cluster-dns="[]"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519231    2582 flags.go:64] FLAG: --cluster-domain=""
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519234    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519237    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519240    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519243    2582 flags.go:64] FLAG: --container-log-max-files="5"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519247    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519250    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519253    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519256    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519259    2582 flags.go:64] FLAG: --contention-profiling="false"
Jul 08 06:54:44.833104 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519262    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519265    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519268    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519271    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519275    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519278    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519281    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519284    2582 flags.go:64] FLAG: --enable-load-reader="false"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519287    2582 flags.go:64] FLAG: --enable-server="true"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519289    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519295    2582 flags.go:64] FLAG: --event-burst="100"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519298    2582 flags.go:64] FLAG: --event-qps="50"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519301    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519304    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519309    2582 flags.go:64] FLAG: --eviction-hard=""
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519312    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519315    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519318    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519321    2582 flags.go:64] FLAG: --eviction-soft=""
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519325    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519327    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519330    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519333    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519336    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519339    2582 flags.go:64] FLAG: --fail-swap-on="true"
Jul 08 06:54:44.833736 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519341    2582 flags.go:64] FLAG: --feature-gates=""
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519345    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519347    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519350    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519353    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519356    2582 flags.go:64] FLAG: --healthz-port="10248"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519359    2582 flags.go:64] FLAG: --help="false"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519361    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-140-110.ec2.internal"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519364    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519367    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519370    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519373    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519376    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519379    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519382    2582 flags.go:64] FLAG: --image-service-endpoint=""
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519384    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519387    2582 flags.go:64] FLAG: --kube-api-burst="100"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519391    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519394    2582 flags.go:64] FLAG: --kube-api-qps="50"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519397    2582 flags.go:64] FLAG: --kube-reserved=""
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519400    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519404    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519407    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519410    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 08 06:54:44.834493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519413    2582 flags.go:64] FLAG: --lock-file=""
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519415    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519419    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519422    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519427    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519429    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519432    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519435    2582 flags.go:64] FLAG: --logging-format="text"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519437    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519440    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519443    2582 flags.go:64] FLAG: --manifest-url=""
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519446    2582 flags.go:64] FLAG: --manifest-url-header=""
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519450    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519453    2582 flags.go:64] FLAG: --max-open-files="1000000"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519457    2582 flags.go:64] FLAG: --max-pods="110"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519459    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519462    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519465    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519468    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519471    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519473    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519476    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519483    2582 flags.go:64] FLAG: --node-status-max-images="50"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519486    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519489    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 08 06:54:44.835120 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519494    2582 flags.go:64] FLAG: --pod-cidr=""
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519496    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519502    2582 flags.go:64] FLAG: --pod-manifest-path=""
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519504    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519509    2582 flags.go:64] FLAG: --pods-per-core="0"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519512    2582 flags.go:64] FLAG: --port="10250"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519515    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519518    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dcf959c24369d56d"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519521    2582 flags.go:64] FLAG: --qos-reserved=""
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519524    2582 flags.go:64] FLAG: --read-only-port="10255"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519527    2582 flags.go:64] FLAG: --register-node="true"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519530    2582 flags.go:64] FLAG: --register-with-taints=""
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519534    2582 flags.go:64] FLAG: --registry-burst="10"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519536    2582 flags.go:64] FLAG: --registry-qps="5"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519539    2582 flags.go:64] FLAG: --reserved-cpus=""
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519542    2582 flags.go:64] FLAG: --reserved-memory=""
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519545    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519548    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519551    2582 flags.go:64] FLAG: --rotate-certificates="false"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519554    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519556    2582 flags.go:64] FLAG: --runonce="false"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519559    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519562    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519565    2582 flags.go:64] FLAG: --seccomp-default="false"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519568    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 08 06:54:44.835843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519570    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519573    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519576    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519579    2582 flags.go:64] FLAG: --storage-driver-password="root"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519582    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519585    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519587    2582 flags.go:64] FLAG: --storage-driver-user="root"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519590    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519595    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519597    2582 flags.go:64] FLAG: --system-cgroups=""
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519600    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519607    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519609    2582 flags.go:64] FLAG: --tls-cert-file=""
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519612    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519616    2582 flags.go:64] FLAG: --tls-min-version=""
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519619    2582 flags.go:64] FLAG: --tls-private-key-file=""
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519622    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519625    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519629    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519632    2582 flags.go:64] FLAG: --v="2"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519635    2582 flags.go:64] FLAG: --version="false"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519639    2582 flags.go:64] FLAG: --vmodule=""
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519643    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.519646    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519755    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 06:54:44.836785 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519759    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519762    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519766    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519769    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519771    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519774    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519776    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519779    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519781    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519784    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519787    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519791    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519794    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519797    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519800    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519804    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519807    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519809    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519814    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519816    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 06:54:44.837501 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519819    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519821    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519824    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519826    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519829    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519831    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519838    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519840    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519843    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519845    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519848    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519850    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519853    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519856    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519858    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519861    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519863    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519866    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519868    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519871    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 06:54:44.921664 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519874    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519876    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519879    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519881    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519884    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519886    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519889    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519893    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519895    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519897    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519902    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519904    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519907    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519909    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519912    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519914    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519917    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519919    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519922    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 06:54:44.922493 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519926    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519928    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519931    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519933    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519936    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519938    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519941    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519943    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519946    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519950    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519953    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519956    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519959    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519961    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519964    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519967    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519969    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519972    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519975    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519977    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 06:54:44.923132 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519982    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519984    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519987    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519991    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519994    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519996    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.519999    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.520003    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.520006    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.520009    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.520012    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.520014    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.520017    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.520020    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.520022    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.520025    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.520027    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.520030    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 06:54:44.923755 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.520032    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 06:54:44.924384 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.520752    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 06:54:44.924384 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.528005    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 08 06:54:44.924384 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.528019    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 08 06:54:44.924384 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.528041    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 08 06:54:44.924384 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.528046    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 08 06:54:44.924384 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528102    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 06:54:44.924384 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528107    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 06:54:44.924384 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528111    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 06:54:44.924384 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528114    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 06:54:44.924384 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528117    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 06:54:44.924384 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528120    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 06:54:44.924384 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528123    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 06:54:44.924384 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528127    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 06:54:44.924384 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528131    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528134    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528137    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528139    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528142    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528145    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528162    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528167    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528170    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528173    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528175    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528178    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528181    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528184    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528186    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528189    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528191    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528194    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528197    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 06:54:44.925009 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528199    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528202    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528205    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528208    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528211    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528213    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528215    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528218    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528220    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528223    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528225    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528228    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528230    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528233    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528235    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528238    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528241    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528244    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528246    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 06:54:44.925640 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528251    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528254    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528258    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528260    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528263    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528266    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528269    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528272    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528274    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528277    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528283    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528286    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528288    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528291    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528294    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528297    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528300    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528303    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528305    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 06:54:44.926476 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528308    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528310    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528313    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528316    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528318    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528321    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528324    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528326    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528329    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528331    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528334    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528336    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528339    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528342    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528344    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528347    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528349    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528352    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528354    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528357    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 06:54:44.927116 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528359    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528362    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528365    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528370    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528373    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528376    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528379    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528384    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528386    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528389    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528392    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528394    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528397    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528399    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.528405    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 06:54:44.927720 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528523    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528527    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528530    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528533    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528537    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528539    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528542    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528545    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528548    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528551    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528554    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528556    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528559    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528562    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528565    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528567    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528570    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528573    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528575    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 06:54:44.928139 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528578    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528583    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528587    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528589    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528592    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528595    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528598    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528601    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528604    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528612    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528615    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528617    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528620    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528622    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528625    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528627    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528630    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528633    2582 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528635    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 06:54:44.928682 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528638    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528640    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528643    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528646    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528648    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528651    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528653    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528656    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528658    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528661    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528663    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528666    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528669    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528671    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528675    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528677    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528680    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528683    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528685    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528689    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 06:54:44.929224 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528692    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528694    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528697    2582 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528700    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528703    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528705    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528708    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528710    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528713    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528715    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528718    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528720    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528724    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528727    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528730    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528733    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528736    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528738    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528741    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 06:54:44.929778 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528743    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528746    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528748    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528751    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528753    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528756    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528758    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528762    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528765    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528767    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528770    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528774    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528778    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528781    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528783    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528786    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528789    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528792    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528795    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 06:54:44.930294 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528797    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 06:54:44.930907 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528800    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 06:54:44.930907 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:44.528803    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 06:54:44.930907 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.528807    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 06:54:44.930907 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.528900    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 08 06:54:44.930907 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.531340    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 08 06:54:44.930907 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.533183    2582 server.go:1019] "Starting client certificate rotation"
Jul 08 06:54:44.930907 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.533282    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 06:54:44.930907 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.533312    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 06:54:44.930907 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.536053    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 06:54:44.930907 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.540394    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 06:54:44.930907 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.554724    2582 log.go:25] "Validated CRI v1 runtime API"
Jul 08 06:54:44.930907 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.557818    2582 log.go:25] "Validated CRI v1 image API"
Jul 08 06:54:44.930907 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.558904    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 08 06:54:44.931317 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.561362    2582 fs.go:135] Filesystem UUIDs: map[686c43a8-8064-4031-995e-4801c44663b8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b7221a11-ca69-4ca2-8dec-a039b1dd35db:/dev/nvme0n1p3]
Jul 08 06:54:44.931317 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.561379    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 08 06:54:44.931317 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.565014    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.566445    2582 manager.go:216] Machine: {Timestamp:2026-07-08 06:54:44.565060438 +0000 UTC m=+0.368846829 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100058 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21b455392f978f7f03b6c759232f60 SystemUUID:ec21b455-392f-978f-7f03-b6c759232f60 BootID:1bfb7993-0d04-4229-b87c-dd3b280f4240 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:97:ab:9e:b0:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:97:ab:9e:b0:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:e4:bb:dd:8c:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.566540    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.566643    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.570221    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.570249    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-110.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.571113    2582 topology_manager.go:138] "Creating topology manager with none policy"
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.571130    2582 container_manager_linux.go:309] "Creating device plugin manager"
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.571140    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.571989    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.573967    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.578778    2582 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.578896    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.578923    2582 kubelet.go:486] "Attempting to sync node with API server"
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.578932    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.578947    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.578959    2582 kubelet.go:398] "Adding apiserver pod source"
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.578967    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 08 06:54:44.931509 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.579317    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.579340    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.581689    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.583988    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.584015    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.585999    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.586014    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.586022    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.586029    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.586041    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.586051    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.586063    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.586069    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.586077    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.586082    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.586091    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.586096    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.586964    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.586973    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.588610    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-110.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.588653    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.588653    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.589280    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kwfht"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.590317    2582 server.go:1268] "Started kubelet"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.590628    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.591568    2582 server.go:310] "Adding debug handlers to kubelet server"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.591583    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.593981    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.596060    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.597707    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kwfht"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.599341    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.602048    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-110.ec2.internal.18c03e0e892840bc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-110.ec2.internal,UID:ip-10-0-140-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-110.ec2.internal,},FirstTimestamp:2026-07-08 06:54:44.590297276 +0000 UTC m=+0.394083667,LastTimestamp:2026-07-08 06:54:44.590297276 +0000 UTC m=+0.394083667,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-110.ec2.internal,}"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.606217    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.606222    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.606297    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.606979    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.607006    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.606979    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.607076    2582 factory.go:55] Registering systemd factory
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.607089    2582 reconstruct.go:97] "Volume reconstruction finished"
Jul 08 06:54:44.932058 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.607097    2582 reconciler.go:29] "Reconciler: start to sync state"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.607115    2582 factory.go:223] Registration of the systemd container factory successfully
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.607916    2582 factory.go:153] Registering CRI-O factory
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.607929    2582 factory.go:223] Registration of the crio container factory successfully
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.607976    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.608002    2582 factory.go:103] Registering Raw factory
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.608017    2582 manager.go:1196] Started watching for new ooms in manager
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.608444    2582 manager.go:349] Starting recovery of all containers
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.616453    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.618582    2582 manager.go:354] Recovery completed
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.619543    2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-110.ec2.internal\" not found" node="ip-10-0-140-110.ec2.internal"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.623377    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.624907    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.624927    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.624936    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.625457    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.625464    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.625468    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.625487    2582 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.626240    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.626266    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.626279    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.627626    2582 policy_none.go:49] "None policy: Start"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.627642    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.627655    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.629642    2582 policy_none.go:47] "Start"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.674720    2582 manager.go:337] "Starting Device Plugin manager"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.674746    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.674754    2582 server.go:87] "Starting device plugin registration server"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.674975    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.674989    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.675092    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.675180    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.675196    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.675622    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.675677    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.738581    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.739751    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.739779    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.739800    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.739837    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.742225    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.775331    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.776050    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.776074    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.776084    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.776104    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-110.ec2.internal"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.785489    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-110.ec2.internal"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.785508    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-110.ec2.internal\": node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:44.933397 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.802021    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.845600    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"]
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.845665    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.846988    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.847010    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.847020    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.849185    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.849238    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.849267    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.849779    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.849797    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.849819    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.849829    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.849803    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.849884    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.852055    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.852077    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.852705    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.852728    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.852737    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.885673    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-110.ec2.internal\" not found" node="ip-10-0-140-110.ec2.internal"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.890866    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-110.ec2.internal\" not found" node="ip-10-0-140-110.ec2.internal"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:44.902905    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.909240    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/813e38510b3ec379e921946926d00ffe-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"813e38510b3ec379e921946926d00ffe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.909285    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f44b43a3e849fcf8ea37cb5848fb430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"7f44b43a3e849fcf8ea37cb5848fb430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:44.934685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:44.909307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f44b43a3e849fcf8ea37cb5848fb430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"7f44b43a3e849fcf8ea37cb5848fb430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:45.032110 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:45.003554    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:45.032110 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.009954    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f44b43a3e849fcf8ea37cb5848fb430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"7f44b43a3e849fcf8ea37cb5848fb430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:45.032110 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.009982    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/813e38510b3ec379e921946926d00ffe-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"813e38510b3ec379e921946926d00ffe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:45.032110 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.010008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f44b43a3e849fcf8ea37cb5848fb430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"7f44b43a3e849fcf8ea37cb5848fb430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:45.032110 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.010054    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f44b43a3e849fcf8ea37cb5848fb430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"7f44b43a3e849fcf8ea37cb5848fb430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:45.032110 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.010066    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/813e38510b3ec379e921946926d00ffe-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"813e38510b3ec379e921946926d00ffe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:45.032110 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.010073    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f44b43a3e849fcf8ea37cb5848fb430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"7f44b43a3e849fcf8ea37cb5848fb430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:45.131585 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:45.104356    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:45.221374 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.188889    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:45.221374 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.193808    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:45.221374 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:45.205078    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:45.335301 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:45.305560    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:45.430510 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:45.406022    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:45.528900 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:45.506828    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:45.646193 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.531803    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 08 06:54:45.646193 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.531927    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 06:54:45.646193 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.531953    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 06:54:45.646193 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.532618    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 06:54:45.646193 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.605993    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 06:49:44 +0000 UTC" deadline="2028-01-26 05:32:19.579292606 +0000 UTC"
Jul 08 06:54:45.646193 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.606019    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13606h37m33.973276519s"
Jul 08 06:54:45.646193 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:45.607202    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:45.725624 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:45.708101    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:45.871485 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:45.808321    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:45.952905 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:45.908991    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found"
Jul 08 06:54:45.960625 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.959093    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 06:54:46.049527 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:45.963652    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f44b43a3e849fcf8ea37cb5848fb430.slice/crio-cd2c1aaf270600ad7acef0a19149f6b4da7cb3cb762747f739bc91f81a12b149 WatchSource:0}: Error finding container cd2c1aaf270600ad7acef0a19149f6b4da7cb3cb762747f739bc91f81a12b149: Status 404 returned error can't find the container with id cd2c1aaf270600ad7acef0a19149f6b4da7cb3cb762747f739bc91f81a12b149
Jul 08 06:54:46.049527 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:45.968939    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 06:54:46.049527 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:45.988519    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813e38510b3ec379e921946926d00ffe.slice/crio-759e17f88cc08a94c5c312332496d061f15a408f02fd80ac02bf4d4064c9f634 WatchSource:0}: Error finding container 759e17f88cc08a94c5c312332496d061f15a408f02fd80ac02bf4d4064c9f634: Status 404 returned error can't find the container with id 759e17f88cc08a94c5c312332496d061f15a408f02fd80ac02bf4d4064c9f634
Jul 08 06:54:46.049527 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.006370    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:46.049527 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.019864    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 06:54:46.049527 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.020706    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"
Jul 08 06:54:46.049527 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.033491    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 06:54:46.535737 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.535700    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 06:54:46.580505 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.580277    2582 apiserver.go:52] "Watching apiserver"
Jul 08 06:54:46.589865 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.589841    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 08 06:54:46.590795 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.590424    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw","openshift-cluster-node-tuning-operator/tuned-26sqr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal","openshift-multus/multus-additional-cni-plugins-242sl","openshift-multus/network-metrics-daemon-h7cff","openshift-network-diagnostics/network-check-target-qq8lz","kube-system/global-pull-secret-syncer-9snpb","openshift-dns/node-resolver-6v9zs","openshift-image-registry/node-ca-2tswp","openshift-multus/multus-ph6dk","openshift-network-operator/iptables-alerter-5glln","openshift-ovn-kubernetes/ovnkube-node-qzsh8","kube-system/konnectivity-agent-gswxv"]
Jul 08 06:54:46.593292 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.593043    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2tswp"
Jul 08 06:54:46.601069 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.597732    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.601069 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.597940    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 08 06:54:46.601069 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.598538    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 08 06:54:46.601069 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.600442    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8v7ct\""
Jul 08 06:54:46.601069 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.600754    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 08 06:54:46.601069 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.600803    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 08 06:54:46.601069 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.600868    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 08 06:54:46.601069 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.600880    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 08 06:54:46.601069 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.601018    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 08 06:54:46.601881 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.601826    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4lg2s\""
Jul 08 06:54:46.602647 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.602189    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 08 06:54:46.603710 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.603512    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 08 06:54:46.603808 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.603783    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.605850 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.605605    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gswxv"
Jul 08 06:54:46.606659 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.606640    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 08 06:54:46.606862 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.606846    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w95z2\""
Jul 08 06:54:46.607060 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.607043    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 08 06:54:46.607291 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.607271    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 08 06:54:46.609103 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.608115    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.609103 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.608239    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 08 06:54:46.609103 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.608449    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hp5zz\""
Jul 08 06:54:46.609103 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.608817    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 08 06:54:46.611061 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.611041    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:46.611273 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:46.611246    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:54:46.612411 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.612390    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c7jfv\""
Jul 08 06:54:46.612511 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.612489    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 08 06:54:46.612660 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.612644    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 08 06:54:46.612734 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.612719    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 08 06:54:46.612861 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.612845    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 08 06:54:46.613340 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.613075    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6v9zs"
Jul 08 06:54:46.613340 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.613138    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 08 06:54:46.616648 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.616626    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 08 06:54:46.616894 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.616877    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 08 06:54:46.617180 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.617163    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-snv62\""
Jul 08 06:54:46.617530 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.617514    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:54:46.617612 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:46.617584    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:54:46.618431 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.618411    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.619733 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1fe61b54-f929-432e-a735-6784ec370581-cnibin\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.619733 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1fe61b54-f929-432e-a735-6784ec370581-os-release\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.619733 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619569    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1fe61b54-f929-432e-a735-6784ec370581-tuning-conf-dir\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.619733 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619597    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-systemd-units\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.619733 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-etc-openvswitch\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.619733 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619649    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-run-ovn\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.619733 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619680    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51fa0f41-b4bf-40c0-bce6-cda5e5629429-ovnkube-config\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.619733 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619710    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51fa0f41-b4bf-40c0-bce6-cda5e5629429-ovnkube-script-lib\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.619733 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619731    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-sys\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.620197 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe61b54-f929-432e-a735-6784ec370581-system-cni-dir\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.620197 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-run-openvswitch\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.620197 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51fa0f41-b4bf-40c0-bce6-cda5e5629429-ovn-node-metrics-cert\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.620197 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619817    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-modprobe-d\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.620197 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-lib-modules\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.620197 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619872    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-var-lib-kubelet\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.620197 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619923    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rw6r\" (UniqueName: \"kubernetes.io/projected/5ae16139-e371-414e-982c-221c0223968a-kube-api-access-6rw6r\") pod \"node-ca-2tswp\" (UID: \"5ae16139-e371-414e-982c-221c0223968a\") " pod="openshift-image-registry/node-ca-2tswp"
Jul 08 06:54:46.620197 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.619969    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-run-netns\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.620197 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620025    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1fe61b54-f929-432e-a735-6784ec370581-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.620197 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1fe61b54-f929-432e-a735-6784ec370581-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.620197 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620100    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd9cj\" (UniqueName: \"kubernetes.io/projected/1fe61b54-f929-432e-a735-6784ec370581-kube-api-access-rd9cj\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.620197 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620126    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-kubernetes\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.620197 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-sysctl-conf\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.620197 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620176    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-sysconfig\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620200    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-systemd\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-run\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a392272-c07e-41e4-997b-af669a4bd16e-agent-certs\") pod \"konnectivity-agent-gswxv\" (UID: \"6a392272-c07e-41e4-997b-af669a4bd16e\") " pod="kube-system/konnectivity-agent-gswxv"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a392272-c07e-41e4-997b-af669a4bd16e-konnectivity-ca\") pod \"konnectivity-agent-gswxv\" (UID: \"6a392272-c07e-41e4-997b-af669a4bd16e\") " pod="kube-system/konnectivity-agent-gswxv"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620315    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-kubelet\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-slash\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-run-systemd\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-node-log\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620504    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620559    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-sysctl-d\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-host\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-cni-bin\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620655    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51fa0f41-b4bf-40c0-bce6-cda5e5629429-env-overrides\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-tuned\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620720    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b8d7da8-5da9-49aa-abea-c9541eff5790-tmp\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.620912 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:46.621950 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1fe61b54-f929-432e-a735-6784ec370581-cni-binary-copy\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.621950 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ae16139-e371-414e-982c-221c0223968a-host\") pod \"node-ca-2tswp\" (UID: \"5ae16139-e371-414e-982c-221c0223968a\") " pod="openshift-image-registry/node-ca-2tswp"
Jul 08 06:54:46.621950 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:46.620784    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:54:46.621950 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ae16139-e371-414e-982c-221c0223968a-serviceca\") pod \"node-ca-2tswp\" (UID: \"5ae16139-e371-414e-982c-221c0223968a\") " pod="openshift-image-registry/node-ca-2tswp"
Jul 08 06:54:46.621950 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620847    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-var-lib-openvswitch\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.621950 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620881    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-log-socket\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.621950 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-cni-netd\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.621950 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4klrz\" (UniqueName: \"kubernetes.io/projected/51fa0f41-b4bf-40c0-bce6-cda5e5629429-kube-api-access-4klrz\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.621950 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.620955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn6qc\" (UniqueName: \"kubernetes.io/projected/9b8d7da8-5da9-49aa-abea-c9541eff5790-kube-api-access-tn6qc\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.621950 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.621472    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 08 06:54:46.621950 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.621764    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-twm4w\""
Jul 08 06:54:46.622516 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.622089    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 08 06:54:46.623870 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.623488    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 08 06:54:46.623870 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.623703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.625971 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.625953    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5glln"
Jul 08 06:54:46.626809 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.626656    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 08 06:54:46.626896 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.626865    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hs7f5\""
Jul 08 06:54:46.627097 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.627012    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 06:54:46.628676 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.628656    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 08 06:54:46.628908 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.628656    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 08 06:54:46.629299 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.629272    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-76mjg\""
Jul 08 06:54:46.629973 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.629953    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 08 06:54:46.643855 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.643835    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 06:54:46.650423 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.650398    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vzsqf"
Jul 08 06:54:46.658941 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.658923    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vzsqf"
Jul 08 06:54:46.708454 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.708422    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 08 06:54:46.721611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721586    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-etc-openvswitch\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.721722 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721628    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-run-ovn\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.721722 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51fa0f41-b4bf-40c0-bce6-cda5e5629429-ovnkube-config\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.721722 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51fa0f41-b4bf-40c0-bce6-cda5e5629429-ovnkube-script-lib\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.721722 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721703    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-etc-openvswitch\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.721937 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721715    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-run-ovn\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.721937 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe61b54-f929-432e-a735-6784ec370581-system-cni-dir\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.721937 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721760    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fe61b54-f929-432e-a735-6784ec370581-system-cni-dir\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.721937 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721776    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-socket-dir\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.721937 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721804    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-device-dir\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.721937 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-run-openvswitch\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.721937 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721863    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51fa0f41-b4bf-40c0-bce6-cda5e5629429-ovn-node-metrics-cert\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.721937 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721891    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-modprobe-d\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.721937 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721898    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-run-openvswitch\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.721937 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-lib-modules\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-var-lib-kubelet\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721969    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn447\" (UniqueName: \"kubernetes.io/projected/d397da7e-ef4a-4e32-a8f3-3888459f52b3-kube-api-access-dn447\") pod \"network-metrics-daemon-h7cff\" (UID: \"d397da7e-ef4a-4e32-a8f3-3888459f52b3\") " pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.721995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-sys-fs\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-hostroot\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-run-netns\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722098    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dbf315f9-c635-4d76-81fb-39beef3d0746-kubelet-config\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxnpz\" (UniqueName: \"kubernetes.io/projected/be1e2414-bf05-480e-91c9-22646d837e75-kube-api-access-bxnpz\") pod \"iptables-alerter-5glln\" (UID: \"be1e2414-bf05-480e-91c9-22646d837e75\") " pod="openshift-network-operator/iptables-alerter-5glln"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722148    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4087521b-c577-40da-9545-89e917d91531-cni-binary-copy\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722174    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4087521b-c577-40da-9545-89e917d91531-multus-daemon-config\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-kubernetes\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-sysctl-conf\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722274    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-sysconfig\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-run\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722297    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-registration-dir\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722359    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-var-lib-kubelet\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.722611 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-kubelet\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-run-systemd\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-node-log\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722454    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51fa0f41-b4bf-40c0-bce6-cda5e5629429-ovnkube-config\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722456    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51fa0f41-b4bf-40c0-bce6-cda5e5629429-ovnkube-script-lib\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-sysctl-d\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722555    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-host\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722575    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-modprobe-d\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722579    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-tuned\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722621    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-cni-bin\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722649    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ae16139-e371-414e-982c-221c0223968a-host\") pod \"node-ca-2tswp\" (UID: \"5ae16139-e371-414e-982c-221c0223968a\") " pod="openshift-image-registry/node-ca-2tswp"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722673    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-sysconfig\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722673    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-var-lib-kubelet\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ae16139-e371-414e-982c-221c0223968a-serviceca\") pod \"node-ca-2tswp\" (UID: \"5ae16139-e371-414e-982c-221c0223968a\") " pod="openshift-image-registry/node-ca-2tswp"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722621    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-kubernetes\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.723475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722780    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs\") pod \"network-metrics-daemon-h7cff\" (UID: \"d397da7e-ef4a-4e32-a8f3-3888459f52b3\") " pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722832    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-sysctl-conf\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722836    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722883    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-run-systemd\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.722940    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-cni-bin\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723028    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-lib-modules\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723067    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-kubelet\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723105    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-node-log\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723145    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/be1e2414-bf05-480e-91c9-22646d837e75-iptables-alerter-script\") pod \"iptables-alerter-5glln\" (UID: \"be1e2414-bf05-480e-91c9-22646d837e75\") " pod="openshift-network-operator/iptables-alerter-5glln"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723183    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-run-netns\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723180    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-run-multus-certs\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723106    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ae16139-e371-414e-982c-221c0223968a-serviceca\") pod \"node-ca-2tswp\" (UID: \"5ae16139-e371-414e-982c-221c0223968a\") " pod="openshift-image-registry/node-ca-2tswp"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723260    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1fe61b54-f929-432e-a735-6784ec370581-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723290    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-run\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1fe61b54-f929-432e-a735-6784ec370581-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-var-lib-openvswitch\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.724083 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4klrz\" (UniqueName: \"kubernetes.io/projected/51fa0f41-b4bf-40c0-bce6-cda5e5629429-kube-api-access-4klrz\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723367    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-sysctl-d\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723358    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-host\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723418    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723471    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-systemd-units\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-sys\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-etc-selinux\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be1e2414-bf05-480e-91c9-22646d837e75-host-slash\") pod \"iptables-alerter-5glln\" (UID: \"be1e2414-bf05-480e-91c9-22646d837e75\") " pod="openshift-network-operator/iptables-alerter-5glln"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-run-netns\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723621    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dbf315f9-c635-4d76-81fb-39beef3d0746-dbus\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723645    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-system-cni-dir\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723670    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-os-release\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9-tmp-dir\") pod \"node-resolver-6v9zs\" (UID: \"8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9\") " pod="openshift-dns/node-resolver-6v9zs"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723726    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-multus-conf-dir\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723771    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-etc-kubernetes\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-cnibin\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf5st\" (UniqueName: \"kubernetes.io/projected/4087521b-c577-40da-9545-89e917d91531-kube-api-access-lf5st\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.724928 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723855    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-systemd\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723882    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a392272-c07e-41e4-997b-af669a4bd16e-agent-certs\") pod \"konnectivity-agent-gswxv\" (UID: \"6a392272-c07e-41e4-997b-af669a4bd16e\") " pod="kube-system/konnectivity-agent-gswxv"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a392272-c07e-41e4-997b-af669a4bd16e-konnectivity-ca\") pod \"konnectivity-agent-gswxv\" (UID: \"6a392272-c07e-41e4-997b-af669a4bd16e\") " pod="kube-system/konnectivity-agent-gswxv"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723917    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1fe61b54-f929-432e-a735-6784ec370581-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723937    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1fe61b54-f929-432e-a735-6784ec370581-cnibin\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723966    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ae16139-e371-414e-982c-221c0223968a-host\") pod \"node-ca-2tswp\" (UID: \"5ae16139-e371-414e-982c-221c0223968a\") " pod="openshift-image-registry/node-ca-2tswp"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723973    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-var-lib-openvswitch\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723965    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdlb6\" (UniqueName: \"kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6\") pod \"network-check-target-qq8lz\" (UID: \"ab22f5b5-5417-4d54-aec8-f5bb631ecd37\") " pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-var-lib-cni-bin\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724017    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-systemd-units\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-slash\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724056    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-sys\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b8d7da8-5da9-49aa-abea-c9541eff5790-tmp\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724097    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1fe61b54-f929-432e-a735-6784ec370581-cni-binary-copy\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.723936    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1fe61b54-f929-432e-a735-6784ec370581-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724200    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-run-k8s-cni-cncf-io\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.725522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724245    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-var-lib-cni-multus\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51fa0f41-b4bf-40c0-bce6-cda5e5629429-env-overrides\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nng8\" (UniqueName: \"kubernetes.io/projected/7ffd10e6-f0b4-486e-8d61-2f040538020d-kube-api-access-4nng8\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-multus-cni-dir\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-multus-socket-dir-parent\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd9cj\" (UniqueName: \"kubernetes.io/projected/1fe61b54-f929-432e-a735-6784ec370581-kube-api-access-rd9cj\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724527    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9-hosts-file\") pod \"node-resolver-6v9zs\" (UID: \"8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9\") " pod="openshift-dns/node-resolver-6v9zs"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sgrw\" (UniqueName: \"kubernetes.io/projected/8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9-kube-api-access-7sgrw\") pod \"node-resolver-6v9zs\" (UID: \"8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9\") " pod="openshift-dns/node-resolver-6v9zs"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-log-socket\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-cni-netd\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724641    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn6qc\" (UniqueName: \"kubernetes.io/projected/9b8d7da8-5da9-49aa-abea-c9541eff5790-kube-api-access-tn6qc\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1fe61b54-f929-432e-a735-6784ec370581-os-release\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724693    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51fa0f41-b4bf-40c0-bce6-cda5e5629429-env-overrides\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724747    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6a392272-c07e-41e4-997b-af669a4bd16e-konnectivity-ca\") pod \"konnectivity-agent-gswxv\" (UID: \"6a392272-c07e-41e4-997b-af669a4bd16e\") " pod="kube-system/konnectivity-agent-gswxv"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1fe61b54-f929-432e-a735-6784ec370581-tuning-conf-dir\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rw6r\" (UniqueName: \"kubernetes.io/projected/5ae16139-e371-414e-982c-221c0223968a-kube-api-access-6rw6r\") pod \"node-ca-2tswp\" (UID: \"5ae16139-e371-414e-982c-221c0223968a\") " pod="openshift-image-registry/node-ca-2tswp"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724828    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-log-socket\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.726452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724898    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1fe61b54-f929-432e-a735-6784ec370581-os-release\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.727300 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.724947    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1fe61b54-f929-432e-a735-6784ec370581-cnibin\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.727300 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.725081    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-systemd\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.727300 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.725121    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-slash\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.727300 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.725393    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1fe61b54-f929-432e-a735-6784ec370581-cni-binary-copy\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.727300 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.725460    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51fa0f41-b4bf-40c0-bce6-cda5e5629429-host-cni-netd\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.727300 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.725463    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1fe61b54-f929-432e-a735-6784ec370581-tuning-conf-dir\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.729017 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.728965    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b8d7da8-5da9-49aa-abea-c9541eff5790-tmp\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.729140 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.729115    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51fa0f41-b4bf-40c0-bce6-cda5e5629429-ovn-node-metrics-cert\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.729662 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.729621    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6a392272-c07e-41e4-997b-af669a4bd16e-agent-certs\") pod \"konnectivity-agent-gswxv\" (UID: \"6a392272-c07e-41e4-997b-af669a4bd16e\") " pod="kube-system/konnectivity-agent-gswxv"
Jul 08 06:54:46.730702 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.730648    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b8d7da8-5da9-49aa-abea-c9541eff5790-etc-tuned\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.742733 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.742668    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4klrz\" (UniqueName: \"kubernetes.io/projected/51fa0f41-b4bf-40c0-bce6-cda5e5629429-kube-api-access-4klrz\") pod \"ovnkube-node-qzsh8\" (UID: \"51fa0f41-b4bf-40c0-bce6-cda5e5629429\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.743351 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.743312    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd9cj\" (UniqueName: \"kubernetes.io/projected/1fe61b54-f929-432e-a735-6784ec370581-kube-api-access-rd9cj\") pod \"multus-additional-cni-plugins-242sl\" (UID: \"1fe61b54-f929-432e-a735-6784ec370581\") " pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.743983 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.743917    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn6qc\" (UniqueName: \"kubernetes.io/projected/9b8d7da8-5da9-49aa-abea-c9541eff5790-kube-api-access-tn6qc\") pod \"tuned-26sqr\" (UID: \"9b8d7da8-5da9-49aa-abea-c9541eff5790\") " pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.744493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.744472    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rw6r\" (UniqueName: \"kubernetes.io/projected/5ae16139-e371-414e-982c-221c0223968a-kube-api-access-6rw6r\") pod \"node-ca-2tswp\" (UID: \"5ae16139-e371-414e-982c-221c0223968a\") " pod="openshift-image-registry/node-ca-2tswp"
Jul 08 06:54:46.745560 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.745455    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" event={"ID":"813e38510b3ec379e921946926d00ffe","Type":"ContainerStarted","Data":"759e17f88cc08a94c5c312332496d061f15a408f02fd80ac02bf4d4064c9f634"}
Jul 08 06:54:46.748640 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.748582    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"7f44b43a3e849fcf8ea37cb5848fb430","Type":"ContainerStarted","Data":"cd2c1aaf270600ad7acef0a19149f6b4da7cb3cb762747f739bc91f81a12b149"}
Jul 08 06:54:46.825667 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.825633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs\") pod \"network-metrics-daemon-h7cff\" (UID: \"d397da7e-ef4a-4e32-a8f3-3888459f52b3\") " pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:46.825815 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.825676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/be1e2414-bf05-480e-91c9-22646d837e75-iptables-alerter-script\") pod \"iptables-alerter-5glln\" (UID: \"be1e2414-bf05-480e-91c9-22646d837e75\") " pod="openshift-network-operator/iptables-alerter-5glln"
Jul 08 06:54:46.825815 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.825703    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-run-multus-certs\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.825815 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.825761    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-run-multus-certs\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.825815 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.825803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.825815 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:46.825810    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:46.826084 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.825836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-etc-selinux\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.826084 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.825861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be1e2414-bf05-480e-91c9-22646d837e75-host-slash\") pod \"iptables-alerter-5glln\" (UID: \"be1e2414-bf05-480e-91c9-22646d837e75\") " pod="openshift-network-operator/iptables-alerter-5glln"
Jul 08 06:54:46.826084 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:46.825898    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs podName:d397da7e-ef4a-4e32-a8f3-3888459f52b3 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:47.325858217 +0000 UTC m=+3.129644596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs") pod "network-metrics-daemon-h7cff" (UID: "d397da7e-ef4a-4e32-a8f3-3888459f52b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:46.826084 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.825909    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be1e2414-bf05-480e-91c9-22646d837e75-host-slash\") pod \"iptables-alerter-5glln\" (UID: \"be1e2414-bf05-480e-91c9-22646d837e75\") " pod="openshift-network-operator/iptables-alerter-5glln"
Jul 08 06:54:46.826084 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.825931    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-run-netns\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826084 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.825927    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.826084 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.825958    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dbf315f9-c635-4d76-81fb-39beef3d0746-dbus\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:46.826084 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.825981    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-etc-selinux\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.826084 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.825997    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-run-netns\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826084 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.825984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-system-cni-dir\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826084 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-os-release\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826084 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9-tmp-dir\") pod \"node-resolver-6v9zs\" (UID: \"8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9\") " pod="openshift-dns/node-resolver-6v9zs"
Jul 08 06:54:46.826084 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826040    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-system-cni-dir\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826084 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826092    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-os-release\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826090    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-multus-conf-dir\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826111    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dbf315f9-c635-4d76-81fb-39beef3d0746-dbus\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826137    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-etc-kubernetes\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826146    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-multus-conf-dir\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-cnibin\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826196    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-etc-kubernetes\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5st\" (UniqueName: \"kubernetes.io/projected/4087521b-c577-40da-9545-89e917d91531-kube-api-access-lf5st\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlb6\" (UniqueName: \"kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6\") pod \"network-check-target-qq8lz\" (UID: \"ab22f5b5-5417-4d54-aec8-f5bb631ecd37\") " pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826253    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-cnibin\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-var-lib-cni-bin\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-run-k8s-cni-cncf-io\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826322    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/be1e2414-bf05-480e-91c9-22646d837e75-iptables-alerter-script\") pod \"iptables-alerter-5glln\" (UID: \"be1e2414-bf05-480e-91c9-22646d837e75\") " pod="openshift-network-operator/iptables-alerter-5glln"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826344    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-var-lib-cni-multus\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.826798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nng8\" (UniqueName: \"kubernetes.io/projected/7ffd10e6-f0b4-486e-8d61-2f040538020d-kube-api-access-4nng8\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.827554 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-multus-cni-dir\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.827554 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-multus-socket-dir-parent\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.827554 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826903    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-multus-socket-dir-parent\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.827554 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:46.826930    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:46.827554 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.826977    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-var-lib-cni-multus\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.827554 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827176    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-multus-cni-dir\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.827554 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827241    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9-tmp-dir\") pod \"node-resolver-6v9zs\" (UID: \"8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9\") " pod="openshift-dns/node-resolver-6v9zs"
Jul 08 06:54:46.827554 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827338    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-run-k8s-cni-cncf-io\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.827554 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827400    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-var-lib-cni-bin\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.827554 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827460    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9-hosts-file\") pod \"node-resolver-6v9zs\" (UID: \"8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9\") " pod="openshift-dns/node-resolver-6v9zs"
Jul 08 06:54:46.827554 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sgrw\" (UniqueName: \"kubernetes.io/projected/8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9-kube-api-access-7sgrw\") pod \"node-resolver-6v9zs\" (UID: \"8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9\") " pod="openshift-dns/node-resolver-6v9zs"
Jul 08 06:54:46.827554 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827537    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9-hosts-file\") pod \"node-resolver-6v9zs\" (UID: \"8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9\") " pod="openshift-dns/node-resolver-6v9zs"
Jul 08 06:54:46.827554 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-socket-dir\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:46.827587    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret podName:dbf315f9-c635-4d76-81fb-39beef3d0746 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:47.327567601 +0000 UTC m=+3.131353995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret") pod "global-pull-secret-syncer-9snpb" (UID: "dbf315f9-c635-4d76-81fb-39beef3d0746") : object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-device-dir\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn447\" (UniqueName: \"kubernetes.io/projected/d397da7e-ef4a-4e32-a8f3-3888459f52b3-kube-api-access-dn447\") pod \"network-metrics-daemon-h7cff\" (UID: \"d397da7e-ef4a-4e32-a8f3-3888459f52b3\") " pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827691    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-socket-dir\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-sys-fs\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-hostroot\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dbf315f9-c635-4d76-81fb-39beef3d0746-kubelet-config\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxnpz\" (UniqueName: \"kubernetes.io/projected/be1e2414-bf05-480e-91c9-22646d837e75-kube-api-access-bxnpz\") pod \"iptables-alerter-5glln\" (UID: \"be1e2414-bf05-480e-91c9-22646d837e75\") " pod="openshift-network-operator/iptables-alerter-5glln"
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4087521b-c577-40da-9545-89e917d91531-cni-binary-copy\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827882    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4087521b-c577-40da-9545-89e917d91531-multus-daemon-config\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827916    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-registration-dir\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.827948    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-var-lib-kubelet\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.828038    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-host-var-lib-kubelet\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.828040    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dbf315f9-c635-4d76-81fb-39beef3d0746-kubelet-config\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:46.828241 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.828122    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-device-dir\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.828975 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.828538    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4087521b-c577-40da-9545-89e917d91531-hostroot\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.828975 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.828611    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-sys-fs\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.828975 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.828621    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ffd10e6-f0b4-486e-8d61-2f040538020d-registration-dir\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.828975 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.828639    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4087521b-c577-40da-9545-89e917d91531-multus-daemon-config\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.828975 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.828934    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4087521b-c577-40da-9545-89e917d91531-cni-binary-copy\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.836414 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:46.836394    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 06:54:46.836503 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:46.836419    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 06:54:46.836503 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:46.836431    2582 projected.go:196] Error preparing data for projected volume kube-api-access-xdlb6 for pod openshift-network-diagnostics/network-check-target-qq8lz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:46.836503 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:46.836491    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6 podName:ab22f5b5-5417-4d54-aec8-f5bb631ecd37 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:47.33647655 +0000 UTC m=+3.140262940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xdlb6" (UniqueName: "kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6") pod "network-check-target-qq8lz" (UID: "ab22f5b5-5417-4d54-aec8-f5bb631ecd37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:46.843417 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.843372    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sgrw\" (UniqueName: \"kubernetes.io/projected/8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9-kube-api-access-7sgrw\") pod \"node-resolver-6v9zs\" (UID: \"8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9\") " pod="openshift-dns/node-resolver-6v9zs"
Jul 08 06:54:46.843807 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.843784    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5st\" (UniqueName: \"kubernetes.io/projected/4087521b-c577-40da-9545-89e917d91531-kube-api-access-lf5st\") pod \"multus-ph6dk\" (UID: \"4087521b-c577-40da-9545-89e917d91531\") " pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.844001 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.843983    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nng8\" (UniqueName: \"kubernetes.io/projected/7ffd10e6-f0b4-486e-8d61-2f040538020d-kube-api-access-4nng8\") pod \"aws-ebs-csi-driver-node-c5fjw\" (UID: \"7ffd10e6-f0b4-486e-8d61-2f040538020d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.844001 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.843990    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxnpz\" (UniqueName: \"kubernetes.io/projected/be1e2414-bf05-480e-91c9-22646d837e75-kube-api-access-bxnpz\") pod \"iptables-alerter-5glln\" (UID: \"be1e2414-bf05-480e-91c9-22646d837e75\") " pod="openshift-network-operator/iptables-alerter-5glln"
Jul 08 06:54:46.844412 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.844393    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn447\" (UniqueName: \"kubernetes.io/projected/d397da7e-ef4a-4e32-a8f3-3888459f52b3-kube-api-access-dn447\") pod \"network-metrics-daemon-h7cff\" (UID: \"d397da7e-ef4a-4e32-a8f3-3888459f52b3\") " pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:46.914802 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.914775    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2tswp"
Jul 08 06:54:46.928772 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.928748    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:54:46.929399 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:46.929361    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae16139_e371_414e_982c_221c0223968a.slice/crio-278caec5ef77df9c7ba2ca13a479ce760b72171fa1a92392303f58d3c6337fab WatchSource:0}: Error finding container 278caec5ef77df9c7ba2ca13a479ce760b72171fa1a92392303f58d3c6337fab: Status 404 returned error can't find the container with id 278caec5ef77df9c7ba2ca13a479ce760b72171fa1a92392303f58d3c6337fab
Jul 08 06:54:46.938756 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.938661    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26sqr"
Jul 08 06:54:46.939094 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:46.939066    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fa0f41_b4bf_40c0_bce6_cda5e5629429.slice/crio-200378fe3a441d13954abf2abe6aa5e2bf7deea010eebfc925066fded0d9818c WatchSource:0}: Error finding container 200378fe3a441d13954abf2abe6aa5e2bf7deea010eebfc925066fded0d9818c: Status 404 returned error can't find the container with id 200378fe3a441d13954abf2abe6aa5e2bf7deea010eebfc925066fded0d9818c
Jul 08 06:54:46.949993 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:46.949968    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8d7da8_5da9_49aa_abea_c9541eff5790.slice/crio-4ad3a5e922b061be36f66e546b07a06bdfb08b0c18c2e98b4720e3700af0fc5c WatchSource:0}: Error finding container 4ad3a5e922b061be36f66e546b07a06bdfb08b0c18c2e98b4720e3700af0fc5c: Status 404 returned error can't find the container with id 4ad3a5e922b061be36f66e546b07a06bdfb08b0c18c2e98b4720e3700af0fc5c
Jul 08 06:54:46.953870 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.953840    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gswxv"
Jul 08 06:54:46.962007 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.961982    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-242sl"
Jul 08 06:54:46.972932 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.972910    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6v9zs"
Jul 08 06:54:46.980837 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.980814    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw"
Jul 08 06:54:46.989770 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.989746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ph6dk"
Jul 08 06:54:46.998622 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:46.998602    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5glln"
Jul 08 06:54:47.332185 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:47.332101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs\") pod \"network-metrics-daemon-h7cff\" (UID: \"d397da7e-ef4a-4e32-a8f3-3888459f52b3\") " pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:47.332346 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:47.332184    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:47.332346 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:47.332327    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:47.332453 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:47.332399    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs podName:d397da7e-ef4a-4e32-a8f3-3888459f52b3 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:48.332382971 +0000 UTC m=+4.136169350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs") pod "network-metrics-daemon-h7cff" (UID: "d397da7e-ef4a-4e32-a8f3-3888459f52b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:47.332453 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:47.332332    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:47.332575 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:47.332542    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret podName:dbf315f9-c635-4d76-81fb-39beef3d0746 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:48.332493527 +0000 UTC m=+4.136279910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret") pod "global-pull-secret-syncer-9snpb" (UID: "dbf315f9-c635-4d76-81fb-39beef3d0746") : object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:47.433494 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:47.433406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlb6\" (UniqueName: \"kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6\") pod \"network-check-target-qq8lz\" (UID: \"ab22f5b5-5417-4d54-aec8-f5bb631ecd37\") " pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:54:47.433666 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:47.433600    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 06:54:47.433666 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:47.433626    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 06:54:47.433666 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:47.433641    2582 projected.go:196] Error preparing data for projected volume kube-api-access-xdlb6 for pod openshift-network-diagnostics/network-check-target-qq8lz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:47.433842 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:47.433710    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6 podName:ab22f5b5-5417-4d54-aec8-f5bb631ecd37 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:48.43368947 +0000 UTC m=+4.237475863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdlb6" (UniqueName: "kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6") pod "network-check-target-qq8lz" (UID: "ab22f5b5-5417-4d54-aec8-f5bb631ecd37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:47.662249 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:47.662125    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 06:49:46 +0000 UTC" deadline="2028-01-27 12:48:17.197282355 +0000 UTC"
Jul 08 06:54:47.662249 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:47.662195    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13637h53m29.535091378s"
Jul 08 06:54:47.740110 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:47.740072    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:47.740289 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:47.740252    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:54:47.753917 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:47.753834    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2tswp" event={"ID":"5ae16139-e371-414e-982c-221c0223968a","Type":"ContainerStarted","Data":"278caec5ef77df9c7ba2ca13a479ce760b72171fa1a92392303f58d3c6337fab"}
Jul 08 06:54:47.757979 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:47.757942    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26sqr" event={"ID":"9b8d7da8-5da9-49aa-abea-c9541eff5790","Type":"ContainerStarted","Data":"4ad3a5e922b061be36f66e546b07a06bdfb08b0c18c2e98b4720e3700af0fc5c"}
Jul 08 06:54:47.762070 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:47.761964    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8" event={"ID":"51fa0f41-b4bf-40c0-bce6-cda5e5629429","Type":"ContainerStarted","Data":"200378fe3a441d13954abf2abe6aa5e2bf7deea010eebfc925066fded0d9818c"}
Jul 08 06:54:48.342347 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:48.342265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:48.342464 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:48.342360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs\") pod \"network-metrics-daemon-h7cff\" (UID: \"d397da7e-ef4a-4e32-a8f3-3888459f52b3\") " pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:48.342464 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:48.342384    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:48.342464 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:48.342452    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret podName:dbf315f9-c635-4d76-81fb-39beef3d0746 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:50.342430308 +0000 UTC m=+6.146216695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret") pod "global-pull-secret-syncer-9snpb" (UID: "dbf315f9-c635-4d76-81fb-39beef3d0746") : object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:48.342632 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:48.342493    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:48.342632 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:48.342547    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs podName:d397da7e-ef4a-4e32-a8f3-3888459f52b3 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:50.342531047 +0000 UTC m=+6.146317428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs") pod "network-metrics-daemon-h7cff" (UID: "d397da7e-ef4a-4e32-a8f3-3888459f52b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:48.358945 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:48.358909    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6fe090_7fa6_4c4c_a339_4f0059c5e6e9.slice/crio-e57a6776bbf4a66322103a63dcbe48ecaaa29a4bcdda54f23d1e8b6065a6d32f WatchSource:0}: Error finding container e57a6776bbf4a66322103a63dcbe48ecaaa29a4bcdda54f23d1e8b6065a6d32f: Status 404 returned error can't find the container with id e57a6776bbf4a66322103a63dcbe48ecaaa29a4bcdda54f23d1e8b6065a6d32f
Jul 08 06:54:48.360025 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:48.359992    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a392272_c07e_41e4_997b_af669a4bd16e.slice/crio-fc7b8eb342266d10e439f89bb4ec93e098cedfef9c1b194a5d774cd95a2f92ee WatchSource:0}: Error finding container fc7b8eb342266d10e439f89bb4ec93e098cedfef9c1b194a5d774cd95a2f92ee: Status 404 returned error can't find the container with id fc7b8eb342266d10e439f89bb4ec93e098cedfef9c1b194a5d774cd95a2f92ee
Jul 08 06:54:48.360890 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:48.360858    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4087521b_c577_40da_9545_89e917d91531.slice/crio-d400e4bfd0e585ef0592224fef7dcb94edd24b5dd7c9f04ec263c3d9272cc52c WatchSource:0}: Error finding container d400e4bfd0e585ef0592224fef7dcb94edd24b5dd7c9f04ec263c3d9272cc52c: Status 404 returned error can't find the container with id d400e4bfd0e585ef0592224fef7dcb94edd24b5dd7c9f04ec263c3d9272cc52c
Jul 08 06:54:48.443814 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:48.443728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlb6\" (UniqueName: \"kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6\") pod \"network-check-target-qq8lz\" (UID: \"ab22f5b5-5417-4d54-aec8-f5bb631ecd37\") " pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:54:48.443989 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:48.443895    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 06:54:48.443989 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:48.443923    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 06:54:48.443989 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:48.443937    2582 projected.go:196] Error preparing data for projected volume kube-api-access-xdlb6 for pod openshift-network-diagnostics/network-check-target-qq8lz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:48.444157 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:48.444037    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6 podName:ab22f5b5-5417-4d54-aec8-f5bb631ecd37 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:50.444014726 +0000 UTC m=+6.247801122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdlb6" (UniqueName: "kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6") pod "network-check-target-qq8lz" (UID: "ab22f5b5-5417-4d54-aec8-f5bb631ecd37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:48.663272 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:48.663235    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 06:49:46 +0000 UTC" deadline="2028-02-16 14:24:48.47505761 +0000 UTC"
Jul 08 06:54:48.663272 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:48.663272    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14119h29m59.811789983s"
Jul 08 06:54:48.742142 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:48.741124    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:54:48.742142 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:48.741278    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:54:48.742142 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:48.741363    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:48.742142 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:48.741489    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:54:48.765392 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:48.765363    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gswxv" event={"ID":"6a392272-c07e-41e4-997b-af669a4bd16e","Type":"ContainerStarted","Data":"fc7b8eb342266d10e439f89bb4ec93e098cedfef9c1b194a5d774cd95a2f92ee"}
Jul 08 06:54:48.767008 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:48.766977    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ph6dk" event={"ID":"4087521b-c577-40da-9545-89e917d91531","Type":"ContainerStarted","Data":"d400e4bfd0e585ef0592224fef7dcb94edd24b5dd7c9f04ec263c3d9272cc52c"}
Jul 08 06:54:48.768822 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:48.768792    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6v9zs" event={"ID":"8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9","Type":"ContainerStarted","Data":"e57a6776bbf4a66322103a63dcbe48ecaaa29a4bcdda54f23d1e8b6065a6d32f"}
Jul 08 06:54:48.770097 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:48.770074    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5glln" event={"ID":"be1e2414-bf05-480e-91c9-22646d837e75","Type":"ContainerStarted","Data":"c23f0f041dd013612416c95c2360344864ddb9e71409e2c7d8557c59c491035b"}
Jul 08 06:54:49.125651 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:54:49.125617    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe61b54_f929_432e_a735_6784ec370581.slice/crio-3601f15077d2f8b4c8791429d2db7a4d6d6c9e4fc3cc9025acdcd63d93c93924 WatchSource:0}: Error finding container 3601f15077d2f8b4c8791429d2db7a4d6d6c9e4fc3cc9025acdcd63d93c93924: Status 404 returned error can't find the container with id 3601f15077d2f8b4c8791429d2db7a4d6d6c9e4fc3cc9025acdcd63d93c93924
Jul 08 06:54:49.741405 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:49.740660    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:49.741405 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:49.741025    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:54:49.784462 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:49.784426    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" event={"ID":"813e38510b3ec379e921946926d00ffe","Type":"ContainerStarted","Data":"cd9bc3299fef6e5a01e9778aa5ecaefaed774fbbd1c3df9cd745235708d4f07d"}
Jul 08 06:54:49.789539 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:49.789508    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2tswp" event={"ID":"5ae16139-e371-414e-982c-221c0223968a","Type":"ContainerStarted","Data":"bf0dd0d8bc38d7093b2ac50c20a03e59bc46bf82a873a2d070128a63a9197969"}
Jul 08 06:54:49.793019 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:49.792990    2582 generic.go:356] "Generic (PLEG): container finished" podID="7f44b43a3e849fcf8ea37cb5848fb430" containerID="daa410c4388627a2bd4b2353d527ca6c7f850d6bdfce27c295a893e93ba00d14" exitCode=0
Jul 08 06:54:49.793132 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:49.793060    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"7f44b43a3e849fcf8ea37cb5848fb430","Type":"ContainerDied","Data":"daa410c4388627a2bd4b2353d527ca6c7f850d6bdfce27c295a893e93ba00d14"}
Jul 08 06:54:49.802536 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:49.802505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw" event={"ID":"7ffd10e6-f0b4-486e-8d61-2f040538020d","Type":"ContainerStarted","Data":"3f25b93af125214ab3679b3e38f984b96f301cde8954dcfc3aa8eece24c47ba2"}
Jul 08 06:54:49.804847 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:49.804793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-242sl" event={"ID":"1fe61b54-f929-432e-a735-6784ec370581","Type":"ContainerStarted","Data":"3601f15077d2f8b4c8791429d2db7a4d6d6c9e4fc3cc9025acdcd63d93c93924"}
Jul 08 06:54:49.821096 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:49.821012    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" podStartSLOduration=3.82099431 podStartE2EDuration="3.82099431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:54:49.802045753 +0000 UTC m=+5.605832154" watchObservedRunningTime="2026-07-08 06:54:49.82099431 +0000 UTC m=+5.624780702"
Jul 08 06:54:50.360279 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:50.360238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs\") pod \"network-metrics-daemon-h7cff\" (UID: \"d397da7e-ef4a-4e32-a8f3-3888459f52b3\") " pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:50.360475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:50.360320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:50.360475 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:50.360410    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:50.360475 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:50.360435    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:50.360625 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:50.360479    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs podName:d397da7e-ef4a-4e32-a8f3-3888459f52b3 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:54.360460053 +0000 UTC m=+10.164246432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs") pod "network-metrics-daemon-h7cff" (UID: "d397da7e-ef4a-4e32-a8f3-3888459f52b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:50.360625 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:50.360499    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret podName:dbf315f9-c635-4d76-81fb-39beef3d0746 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:54.360489563 +0000 UTC m=+10.164275943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret") pod "global-pull-secret-syncer-9snpb" (UID: "dbf315f9-c635-4d76-81fb-39beef3d0746") : object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:50.461348 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:50.461313    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlb6\" (UniqueName: \"kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6\") pod \"network-check-target-qq8lz\" (UID: \"ab22f5b5-5417-4d54-aec8-f5bb631ecd37\") " pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:54:50.461519 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:50.461479    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 06:54:50.461519 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:50.461502    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 06:54:50.461519 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:50.461514    2582 projected.go:196] Error preparing data for projected volume kube-api-access-xdlb6 for pod openshift-network-diagnostics/network-check-target-qq8lz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:50.461662 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:50.461574    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6 podName:ab22f5b5-5417-4d54-aec8-f5bb631ecd37 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:54.461556068 +0000 UTC m=+10.265342450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdlb6" (UniqueName: "kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6") pod "network-check-target-qq8lz" (UID: "ab22f5b5-5417-4d54-aec8-f5bb631ecd37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:50.742576 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:50.742542    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:54:50.743011 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:50.742666    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:54:50.743085 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:50.743027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:50.743207 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:50.743124    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:54:51.741209 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:51.740715    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:51.741209 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:51.740850    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:54:52.741167 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:52.741122    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:52.741620 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:52.741279    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:54:52.741775 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:52.741756    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:54:52.741874 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:52.741853    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:54:53.740466 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:53.740435    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:53.740657 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:53.740566    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:54:54.393447 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:54.393360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs\") pod \"network-metrics-daemon-h7cff\" (UID: \"d397da7e-ef4a-4e32-a8f3-3888459f52b3\") " pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:54.393917 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:54.393539    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:54.393917 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:54.393562    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:54.393917 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:54.393610    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs podName:d397da7e-ef4a-4e32-a8f3-3888459f52b3 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:02.393589969 +0000 UTC m=+18.197376367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs") pod "network-metrics-daemon-h7cff" (UID: "d397da7e-ef4a-4e32-a8f3-3888459f52b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:54.393917 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:54.393683    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:54.393917 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:54.393741    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret podName:dbf315f9-c635-4d76-81fb-39beef3d0746 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:02.393724495 +0000 UTC m=+18.197510890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret") pod "global-pull-secret-syncer-9snpb" (UID: "dbf315f9-c635-4d76-81fb-39beef3d0746") : object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:54.494799 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:54.494762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlb6\" (UniqueName: \"kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6\") pod \"network-check-target-qq8lz\" (UID: \"ab22f5b5-5417-4d54-aec8-f5bb631ecd37\") " pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:54:54.494966 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:54.494949    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 06:54:54.495049 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:54.494971    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 06:54:54.495049 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:54.494984    2582 projected.go:196] Error preparing data for projected volume kube-api-access-xdlb6 for pod openshift-network-diagnostics/network-check-target-qq8lz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:54.495173 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:54.495053    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6 podName:ab22f5b5-5417-4d54-aec8-f5bb631ecd37 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:02.495031132 +0000 UTC m=+18.298817515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdlb6" (UniqueName: "kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6") pod "network-check-target-qq8lz" (UID: "ab22f5b5-5417-4d54-aec8-f5bb631ecd37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:54.746341 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:54.743073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:54.746341 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:54.743261    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:54:54.746341 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:54.743782    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:54:54.746341 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:54.743925    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:54:55.740579 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:55.740547    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:55.741084 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:55.740676    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:54:56.740232 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:56.740197    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:56.740366 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:56.740344    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:54:56.740834 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:56.740660    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:54:56.740834 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:56.740781    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:54:57.740091 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:57.740005    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:57.740244 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:57.740120    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:54:58.743691 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:58.743664    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:54:58.744137 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:58.743820    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:54:58.744137 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:58.743851    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:54:58.744137 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:58.743945    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:54:59.740250 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:54:59.740220    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:54:59.740406 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:54:59.740311    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:55:00.743098 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:00.743072    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:00.743098 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:00.743088    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:00.743694 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:00.743198    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:55:00.743694 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:00.743287    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:55:01.740635 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:01.740605    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:01.740849 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:01.740726    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:55:02.457637 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:02.457606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs\") pod \"network-metrics-daemon-h7cff\" (UID: \"d397da7e-ef4a-4e32-a8f3-3888459f52b3\") " pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:02.458052 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:02.457667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:02.458052 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:02.457786    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 06:55:02.458052 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:02.457792    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:55:02.458052 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:02.457844    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret podName:dbf315f9-c635-4d76-81fb-39beef3d0746 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:18.457826718 +0000 UTC m=+34.261613106 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret") pod "global-pull-secret-syncer-9snpb" (UID: "dbf315f9-c635-4d76-81fb-39beef3d0746") : object "kube-system"/"original-pull-secret" not registered
Jul 08 06:55:02.458052 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:02.457869    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs podName:d397da7e-ef4a-4e32-a8f3-3888459f52b3 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:18.457856145 +0000 UTC m=+34.261642534 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs") pod "network-metrics-daemon-h7cff" (UID: "d397da7e-ef4a-4e32-a8f3-3888459f52b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:55:02.558373 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:02.558338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlb6\" (UniqueName: \"kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6\") pod \"network-check-target-qq8lz\" (UID: \"ab22f5b5-5417-4d54-aec8-f5bb631ecd37\") " pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:02.558518 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:02.558496    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 06:55:02.558518 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:02.558512    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 06:55:02.558626 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:02.558521    2582 projected.go:196] Error preparing data for projected volume kube-api-access-xdlb6 for pod openshift-network-diagnostics/network-check-target-qq8lz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:55:02.558626 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:02.558582    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6 podName:ab22f5b5-5417-4d54-aec8-f5bb631ecd37 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:18.558562413 +0000 UTC m=+34.362348802 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdlb6" (UniqueName: "kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6") pod "network-check-target-qq8lz" (UID: "ab22f5b5-5417-4d54-aec8-f5bb631ecd37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:55:02.744286 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:02.744197    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:02.744442 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:02.744321    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:55:02.744442 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:02.744384    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:02.744561 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:02.744491    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:55:03.740114 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:03.740076    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:03.740597 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:03.740241    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:55:04.740906 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:04.740869    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:04.741369 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:04.740983    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:55:04.741369 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:04.741044    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:04.741369 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:04.741177    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:55:05.740225 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.739973    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:05.740349 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:05.740328    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:55:05.843989 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.843960    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"7f44b43a3e849fcf8ea37cb5848fb430","Type":"ContainerStarted","Data":"0975e9f8b26d035dfdc299e9de21093d0acddbf43df436fcd5ddb6e475a95977"}
Jul 08 06:55:05.845760 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.845731    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26sqr" event={"ID":"9b8d7da8-5da9-49aa-abea-c9541eff5790","Type":"ContainerStarted","Data":"f12b60b21eddebbec49b348a28230fa2a7c85f7ffee225880afb73eae7294859"}
Jul 08 06:55:05.848106 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.848086    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8" event={"ID":"51fa0f41-b4bf-40c0-bce6-cda5e5629429","Type":"ContainerStarted","Data":"c14bb557600c766f1ea8e572f581e01666ca6e0d6ff599397879794469502b6c"}
Jul 08 06:55:05.848222 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.848170    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8" event={"ID":"51fa0f41-b4bf-40c0-bce6-cda5e5629429","Type":"ContainerStarted","Data":"92cdf6f1d8cc60a3d15a5e6693fe1854bdcd1153108d6809c7976d471392a377"}
Jul 08 06:55:05.848279 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.848222    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8" event={"ID":"51fa0f41-b4bf-40c0-bce6-cda5e5629429","Type":"ContainerStarted","Data":"7e14439b7de8757b43a8e839a94ec51f9a93ee2d7006b7857bf1690186de1b7d"}
Jul 08 06:55:05.849486 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.849464    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gswxv" event={"ID":"6a392272-c07e-41e4-997b-af669a4bd16e","Type":"ContainerStarted","Data":"fb9cb11f85abce60f30908e9112eaac3949867a1f03e14c542083bb00c762f7c"}
Jul 08 06:55:05.850751 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.850725    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw" event={"ID":"7ffd10e6-f0b4-486e-8d61-2f040538020d","Type":"ContainerStarted","Data":"ff31af9a6f9676cb35cedca755eb87b700dba19d684130db0a026b6ad1794b08"}
Jul 08 06:55:05.852031 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.852010    2582 generic.go:356] "Generic (PLEG): container finished" podID="1fe61b54-f929-432e-a735-6784ec370581" containerID="8004649deba8b66fcc71caf85cdb25523d9e90e058d11e14b8cd0002f61ede5a" exitCode=0
Jul 08 06:55:05.852110 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.852075    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-242sl" event={"ID":"1fe61b54-f929-432e-a735-6784ec370581","Type":"ContainerDied","Data":"8004649deba8b66fcc71caf85cdb25523d9e90e058d11e14b8cd0002f61ede5a"}
Jul 08 06:55:05.853477 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.853459    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ph6dk" event={"ID":"4087521b-c577-40da-9545-89e917d91531","Type":"ContainerStarted","Data":"e7069c1d8dbb4645412d788090c55229846c57645593d3102db2e42bf2c0df5b"}
Jul 08 06:55:05.854870 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.854854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6v9zs" event={"ID":"8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9","Type":"ContainerStarted","Data":"cf4b7de39a702aaee60fef7b1db7262c813e31e753f994d202001fa43f33f88e"}
Jul 08 06:55:05.860846 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.860809    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" podStartSLOduration=19.860797771 podStartE2EDuration="19.860797771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:05.860575502 +0000 UTC m=+21.664361904" watchObservedRunningTime="2026-07-08 06:55:05.860797771 +0000 UTC m=+21.664584170"
Jul 08 06:55:05.861110 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.861086    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2tswp" podStartSLOduration=18.61013409 podStartE2EDuration="20.861077934s" totalImagesPullingTime="2.250943844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:54:49.845558681 +0000 UTC m=+5.649345097" watchObservedRunningTime="2026-07-08 06:55:05.861077934 +0000 UTC m=+21.664864333"
Jul 08 06:55:05.880287 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.880248    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-26sqr" podStartSLOduration=2.377741869 podStartE2EDuration="20.88023801s" totalImagesPullingTime="18.502496141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:05.879995845 +0000 UTC m=+21.683782264" watchObservedRunningTime="2026-07-08 06:55:05.88023801 +0000 UTC m=+21.684024409"
Jul 08 06:55:05.902937 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.902902    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6v9zs" podStartSLOduration=3.80450337 podStartE2EDuration="20.90288693s" totalImagesPullingTime="17.09838356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:05.902568568 +0000 UTC m=+21.706354968" watchObservedRunningTime="2026-07-08 06:55:05.90288693 +0000 UTC m=+21.706673331"
Jul 08 06:55:05.930178 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.930118    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ph6dk" podStartSLOduration=3.7860513129999998 podStartE2EDuration="20.930103135s" totalImagesPullingTime="17.144051822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:05.929822774 +0000 UTC m=+21.733609193" watchObservedRunningTime="2026-07-08 06:55:05.930103135 +0000 UTC m=+21.733889555"
Jul 08 06:55:05.948351 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:05.948313    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gswxv" podStartSLOduration=3.852993017 podStartE2EDuration="20.948297725s" totalImagesPullingTime="17.095304708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:05.947731593 +0000 UTC m=+21.751518012" watchObservedRunningTime="2026-07-08 06:55:05.948297725 +0000 UTC m=+21.752084126"
Jul 08 06:55:06.072667 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:06.072638    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:06.136895 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:06.136865    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:06.740654 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:06.740625    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:06.740878 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:06.740625    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:06.740878 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:06.740743    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:55:06.740878 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:06.740839    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:55:06.860516 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:06.860427    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8" event={"ID":"51fa0f41-b4bf-40c0-bce6-cda5e5629429","Type":"ContainerStarted","Data":"05c04c9edab61f06261a172fa388a25d61e9d805cc3f9910941e6aa21e2d9c9c"}
Jul 08 06:55:06.860863 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:06.860515    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8" event={"ID":"51fa0f41-b4bf-40c0-bce6-cda5e5629429","Type":"ContainerStarted","Data":"c4cb58431f341a0578b07cad2e0adb5eb62896df774d8c419969de839a42b1ee"}
Jul 08 06:55:06.860863 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:06.860571    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8" event={"ID":"51fa0f41-b4bf-40c0-bce6-cda5e5629429","Type":"ContainerStarted","Data":"9a12a9c37c3a6e2115370c4dc170fe9e396883971fe134854a26cd3b910871fe"}
Jul 08 06:55:06.862361 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:06.862325    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5glln" event={"ID":"be1e2414-bf05-480e-91c9-22646d837e75","Type":"ContainerStarted","Data":"b125ba4cef8aa8357a8cfc437856ee00061098c213b52d3dc5f81ab4705c3f9c"}
Jul 08 06:55:06.882299 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:06.882259    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5glln" podStartSLOduration=4.785194971 podStartE2EDuration="21.882243783s" totalImagesPullingTime="17.097048812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:06.881652591 +0000 UTC m=+22.685438991" watchObservedRunningTime="2026-07-08 06:55:06.882243783 +0000 UTC m=+22.686030183"
Jul 08 06:55:06.901353 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:06.901245    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 08 06:55:07.149456 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:07.149381    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:07.236544 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:07.236511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:07.365309 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:07.365276    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:07.606340 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:07.606310    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gswxv"
Jul 08 06:55:07.607162 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:07.607115    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gswxv"
Jul 08 06:55:07.684325 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:07.684219    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T06:55:06.901262393Z","UUID":"b8e533de-bf24-460a-8c04-d0bee1e9b7e4","Handler":null,"Name":"","Endpoint":""}
Jul 08 06:55:07.686017 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:07.685992    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 08 06:55:07.686175 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:07.686038    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 08 06:55:07.740053 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:07.740029    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:07.740214 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:07.740179    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:55:07.867347 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:07.867271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw" event={"ID":"7ffd10e6-f0b4-486e-8d61-2f040538020d","Type":"ContainerStarted","Data":"9baf1868db859a84b3a8e611535256a90ae3640a506ba4bff9c9d88c5ae92ed1"}
Jul 08 06:55:07.867802 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:07.867439    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gswxv"
Jul 08 06:55:07.868021 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:07.868005    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gswxv"
Jul 08 06:55:07.949114 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:07.949077    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:08.182079 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:08.182044    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:08.741121 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:08.741089    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:08.741463 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:08.741089    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:08.741518 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:08.741270    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:55:08.741518 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:08.741345    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:55:08.876167 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:08.876125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8" event={"ID":"51fa0f41-b4bf-40c0-bce6-cda5e5629429","Type":"ContainerStarted","Data":"d85a49bc007ca116be2c9c3a51a44bb96ea915a5e44be123732318dfbc1b1e27"}
Jul 08 06:55:08.878455 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:08.878412    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw" event={"ID":"7ffd10e6-f0b4-486e-8d61-2f040538020d","Type":"ContainerStarted","Data":"827172a5e4802636e3f3187c85b2a515d812992bd0d1fe2cc3ac8f85c7f79f84"}
Jul 08 06:55:08.899794 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:08.899743    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5fjw" podStartSLOduration=5.29547754 podStartE2EDuration="23.899730021s" totalImagesPullingTime="18.604252481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:08.89956486 +0000 UTC m=+24.703351260" watchObservedRunningTime="2026-07-08 06:55:08.899730021 +0000 UTC m=+24.703516421"
Jul 08 06:55:09.741041 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:09.741011    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:09.741251 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:09.741123    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:55:10.740387 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:10.740206    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:10.741198 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:10.740280    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:10.741198 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:10.740488    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:55:10.741198 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:10.740559    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:55:10.883975 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:10.883942    2582 generic.go:356] "Generic (PLEG): container finished" podID="1fe61b54-f929-432e-a735-6784ec370581" containerID="19e37d14f30357c7d2ac96755281983191b20a66c8ad0d09d6cfb69641fd9497" exitCode=0
Jul 08 06:55:10.884171 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:10.884035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-242sl" event={"ID":"1fe61b54-f929-432e-a735-6784ec370581","Type":"ContainerDied","Data":"19e37d14f30357c7d2ac96755281983191b20a66c8ad0d09d6cfb69641fd9497"}
Jul 08 06:55:10.887801 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:10.887765    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8" event={"ID":"51fa0f41-b4bf-40c0-bce6-cda5e5629429","Type":"ContainerStarted","Data":"e14a4e58dc758ef4fe719a11f8a5708171c6d70c7eabb86f338c700b00925cc2"}
Jul 08 06:55:10.888020 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:10.888002    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:55:10.888102 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:10.888075    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:55:10.888170 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:10.888119    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:55:10.903775 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:10.903750    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:55:10.903880 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:10.903871    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:55:10.935299 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:10.935264    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8" podStartSLOduration=7.345456749 podStartE2EDuration="25.935252895s" totalImagesPullingTime="18.589796146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:10.934874118 +0000 UTC m=+26.738660520" watchObservedRunningTime="2026-07-08 06:55:10.935252895 +0000 UTC m=+26.739039294"
Jul 08 06:55:11.740846 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:11.740812    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:11.741399 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:11.740935    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:55:12.312987 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:12.312962    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:12.381874 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:12.381840    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:12.434892 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:12.434867    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9snpb"]
Jul 08 06:55:12.435044 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:12.434957    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:12.435114 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:12.435057    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:55:12.435537 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:12.435508    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h7cff"]
Jul 08 06:55:12.435628 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:12.435601    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:12.435698 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:12.435679    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:55:12.438247 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:12.438224    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qq8lz"]
Jul 08 06:55:12.438319 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:12.438310    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:12.438414 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:12.438395    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:55:12.895052 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:12.895015    2582 generic.go:356] "Generic (PLEG): container finished" podID="1fe61b54-f929-432e-a735-6784ec370581" containerID="98a5a236260839259f661c599d3436955bc0a766ace02ca99b79d5592b1996e8" exitCode=0
Jul 08 06:55:12.895530 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:12.895098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-242sl" event={"ID":"1fe61b54-f929-432e-a735-6784ec370581","Type":"ContainerDied","Data":"98a5a236260839259f661c599d3436955bc0a766ace02ca99b79d5592b1996e8"}
Jul 08 06:55:13.740774 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:13.740610    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:13.740920 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:13.740666    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:13.740920 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:13.740876    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:55:13.740990 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:13.740922    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:55:14.742128 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:14.742100    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:14.742596 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:14.742219    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:55:14.900934 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:14.900903    2582 generic.go:356] "Generic (PLEG): container finished" podID="1fe61b54-f929-432e-a735-6784ec370581" containerID="4f0919872defcab361f664f108e7adfe82db27b34c0882328837f9d59825ffdb" exitCode=0
Jul 08 06:55:14.901075 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:14.900952    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-242sl" event={"ID":"1fe61b54-f929-432e-a735-6784ec370581","Type":"ContainerDied","Data":"4f0919872defcab361f664f108e7adfe82db27b34c0882328837f9d59825ffdb"}
Jul 08 06:55:15.740293 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:15.740260    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:15.740467 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:15.740298    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:15.740467 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:15.740389    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:55:15.740562 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:15.740544    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:55:16.356537 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:16.356500    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:16.453931 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:16.453883    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:16.740910 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:16.740879    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:16.741078 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:16.741005    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq8lz" podUID="ab22f5b5-5417-4d54-aec8-f5bb631ecd37"
Jul 08 06:55:17.426167 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:17.426127    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:17.491546 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:17.491515    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:17.740560 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:17.740487    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:17.740725 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:17.740491    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:17.740725 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:17.740616    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9snpb" podUID="dbf315f9-c635-4d76-81fb-39beef3d0746"
Jul 08 06:55:17.740725 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:17.740688    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7cff" podUID="d397da7e-ef4a-4e32-a8f3-3888459f52b3"
Jul 08 06:55:18.219498 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.219457    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:18.287984 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.287948    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:18.477805 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.477736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs\") pod \"network-metrics-daemon-h7cff\" (UID: \"d397da7e-ef4a-4e32-a8f3-3888459f52b3\") " pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:18.477805 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.477783    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:18.478247 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:18.477888    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 06:55:18.478247 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:18.477892    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:55:18.478247 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:18.477961    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret podName:dbf315f9-c635-4d76-81fb-39beef3d0746 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:50.477940533 +0000 UTC m=+66.281726912 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret") pod "global-pull-secret-syncer-9snpb" (UID: "dbf315f9-c635-4d76-81fb-39beef3d0746") : object "kube-system"/"original-pull-secret" not registered
Jul 08 06:55:18.478247 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:18.477981    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs podName:d397da7e-ef4a-4e32-a8f3-3888459f52b3 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:50.477971281 +0000 UTC m=+66.281757658 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs") pod "network-metrics-daemon-h7cff" (UID: "d397da7e-ef4a-4e32-a8f3-3888459f52b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:55:18.506124 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.506094    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeReady"
Jul 08 06:55:18.506285 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.506270    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 08 06:55:18.558141 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.558111    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gjw5t"]
Jul 08 06:55:18.563261 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.563242    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:18.568867 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.568216    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wxcjq"]
Jul 08 06:55:18.569125 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.569105    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6zr5\""
Jul 08 06:55:18.569233 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.569172    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 08 06:55:18.569322 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.569300    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 08 06:55:18.572439 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.572420    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wxcjq"
Jul 08 06:55:18.576220 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.576199    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gjw5t"]
Jul 08 06:55:18.576325 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.576231    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 08 06:55:18.576325 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.576312    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 08 06:55:18.576635 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.576606    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 08 06:55:18.576796 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.576611    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fs5kh\""
Jul 08 06:55:18.578081 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.578061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlb6\" (UniqueName: \"kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6\") pod \"network-check-target-qq8lz\" (UID: \"ab22f5b5-5417-4d54-aec8-f5bb631ecd37\") " pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:18.578291 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:18.578271    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 06:55:18.578390 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:18.578300    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 06:55:18.578390 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:18.578313    2582 projected.go:196] Error preparing data for projected volume kube-api-access-xdlb6 for pod openshift-network-diagnostics/network-check-target-qq8lz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:55:18.578390 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:18.578383    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6 podName:ab22f5b5-5417-4d54-aec8-f5bb631ecd37 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:50.578364298 +0000 UTC m=+66.382150676 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdlb6" (UniqueName: "kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6") pod "network-check-target-qq8lz" (UID: "ab22f5b5-5417-4d54-aec8-f5bb631ecd37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:55:18.583027 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.583006    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wxcjq"]
Jul 08 06:55:18.678991 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.678946    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:18.678991 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.678989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm4nz\" (UniqueName: \"kubernetes.io/projected/e142a9e3-7af5-4e3a-9b0e-070f68044486-kube-api-access-tm4nz\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:18.679232 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.679074    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmgzj\" (UniqueName: \"kubernetes.io/projected/bc6bc5da-ac3a-43be-a3b9-7468831da20e-kube-api-access-wmgzj\") pod \"ingress-canary-wxcjq\" (UID: \"bc6bc5da-ac3a-43be-a3b9-7468831da20e\") " pod="openshift-ingress-canary/ingress-canary-wxcjq"
Jul 08 06:55:18.679232 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.679141    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc6bc5da-ac3a-43be-a3b9-7468831da20e-cert\") pod \"ingress-canary-wxcjq\" (UID: \"bc6bc5da-ac3a-43be-a3b9-7468831da20e\") " pod="openshift-ingress-canary/ingress-canary-wxcjq"
Jul 08 06:55:18.679232 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.679220    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e142a9e3-7af5-4e3a-9b0e-070f68044486-tmp-dir\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:18.679380 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.679249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e142a9e3-7af5-4e3a-9b0e-070f68044486-config-volume\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:18.720627 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.720597    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:18.740693 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.740650    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:18.743425 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.743395    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 06:55:18.743516 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.743451    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 06:55:18.743516 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.743452    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6n2tj\""
Jul 08 06:55:18.779601 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.779584    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e142a9e3-7af5-4e3a-9b0e-070f68044486-tmp-dir\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:18.779685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.779610    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e142a9e3-7af5-4e3a-9b0e-070f68044486-config-volume\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:18.779685 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.779667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:18.779856 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.779705    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm4nz\" (UniqueName: \"kubernetes.io/projected/e142a9e3-7af5-4e3a-9b0e-070f68044486-kube-api-access-tm4nz\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:18.779856 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.779767    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmgzj\" (UniqueName: \"kubernetes.io/projected/bc6bc5da-ac3a-43be-a3b9-7468831da20e-kube-api-access-wmgzj\") pod \"ingress-canary-wxcjq\" (UID: \"bc6bc5da-ac3a-43be-a3b9-7468831da20e\") " pod="openshift-ingress-canary/ingress-canary-wxcjq"
Jul 08 06:55:18.779856 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.779792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc6bc5da-ac3a-43be-a3b9-7468831da20e-cert\") pod \"ingress-canary-wxcjq\" (UID: \"bc6bc5da-ac3a-43be-a3b9-7468831da20e\") " pod="openshift-ingress-canary/ingress-canary-wxcjq"
Jul 08 06:55:18.779856 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:18.779795    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:55:18.780038 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:18.779887    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls podName:e142a9e3-7af5-4e3a-9b0e-070f68044486 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:19.279870592 +0000 UTC m=+35.083656970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls") pod "dns-default-gjw5t" (UID: "e142a9e3-7af5-4e3a-9b0e-070f68044486") : secret "dns-default-metrics-tls" not found
Jul 08 06:55:18.780038 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.779984    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e142a9e3-7af5-4e3a-9b0e-070f68044486-tmp-dir\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:18.780351 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.780332    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e142a9e3-7af5-4e3a-9b0e-070f68044486-config-volume\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:18.784097 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.783975    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc6bc5da-ac3a-43be-a3b9-7468831da20e-cert\") pod \"ingress-canary-wxcjq\" (UID: \"bc6bc5da-ac3a-43be-a3b9-7468831da20e\") " pod="openshift-ingress-canary/ingress-canary-wxcjq"
Jul 08 06:55:18.789844 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.789816    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmgzj\" (UniqueName: \"kubernetes.io/projected/bc6bc5da-ac3a-43be-a3b9-7468831da20e-kube-api-access-wmgzj\") pod \"ingress-canary-wxcjq\" (UID: \"bc6bc5da-ac3a-43be-a3b9-7468831da20e\") " pod="openshift-ingress-canary/ingress-canary-wxcjq"
Jul 08 06:55:18.789954 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.789842    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm4nz\" (UniqueName: \"kubernetes.io/projected/e142a9e3-7af5-4e3a-9b0e-070f68044486-kube-api-access-tm4nz\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:18.826492 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.826467    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:18.884190 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.884166    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wxcjq"
Jul 08 06:55:18.915393 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:18.915364    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-242sl" event={"ID":"1fe61b54-f929-432e-a735-6784ec370581","Type":"ContainerStarted","Data":"a99b96f72fd3e4eb01ea1a60978f4d10dddf8b36257bb6250b81da6777c28caf"}
Jul 08 06:55:19.019765 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:19.019693    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wxcjq"]
Jul 08 06:55:19.024849 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:55:19.024821    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6bc5da_ac3a_43be_a3b9_7468831da20e.slice/crio-ac1889f5ac87d7b23d80ccceb59edbb2327f5bdb9b31f267d3e6dd1d3fd75d2c WatchSource:0}: Error finding container ac1889f5ac87d7b23d80ccceb59edbb2327f5bdb9b31f267d3e6dd1d3fd75d2c: Status 404 returned error can't find the container with id ac1889f5ac87d7b23d80ccceb59edbb2327f5bdb9b31f267d3e6dd1d3fd75d2c
Jul 08 06:55:19.282620 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:19.282554    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:19.282726 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:19.282663    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:55:19.282726 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:19.282714    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls podName:e142a9e3-7af5-4e3a-9b0e-070f68044486 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:20.282696823 +0000 UTC m=+36.086483204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls") pod "dns-default-gjw5t" (UID: "e142a9e3-7af5-4e3a-9b0e-070f68044486") : secret "dns-default-metrics-tls" not found
Jul 08 06:55:19.580346 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:19.580268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:19.647675 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:19.647642    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:19.740863 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:19.740830    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:19.741018 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:19.740845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:19.744128 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:19.744105    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 06:55:19.744259 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:19.744199    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 06:55:19.744341 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:19.744331    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4f688\""
Jul 08 06:55:19.921144 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:19.921108    2582 generic.go:356] "Generic (PLEG): container finished" podID="1fe61b54-f929-432e-a735-6784ec370581" containerID="a99b96f72fd3e4eb01ea1a60978f4d10dddf8b36257bb6250b81da6777c28caf" exitCode=0
Jul 08 06:55:19.921309 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:19.921192    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-242sl" event={"ID":"1fe61b54-f929-432e-a735-6784ec370581","Type":"ContainerDied","Data":"a99b96f72fd3e4eb01ea1a60978f4d10dddf8b36257bb6250b81da6777c28caf"}
Jul 08 06:55:19.922460 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:19.922438    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wxcjq" event={"ID":"bc6bc5da-ac3a-43be-a3b9-7468831da20e","Type":"ContainerStarted","Data":"ac1889f5ac87d7b23d80ccceb59edbb2327f5bdb9b31f267d3e6dd1d3fd75d2c"}
Jul 08 06:55:20.290795 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.290581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:20.290948 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:20.290733    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:55:20.290948 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:20.290906    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls podName:e142a9e3-7af5-4e3a-9b0e-070f68044486 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:22.290882511 +0000 UTC m=+38.094668900 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls") pod "dns-default-gjw5t" (UID: "e142a9e3-7af5-4e3a-9b0e-070f68044486") : secret "dns-default-metrics-tls" not found
Jul 08 06:55:20.596545 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.596465    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qb2c9"]
Jul 08 06:55:20.601162 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.601132    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qb2c9"
Jul 08 06:55:20.604940 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.604918    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lspz6\""
Jul 08 06:55:20.606438 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.606396    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 08 06:55:20.606956 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.606936    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 08 06:55:20.612984 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.612961    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qb2c9"]
Jul 08 06:55:20.693609 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.693580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4nt\" (UniqueName: \"kubernetes.io/projected/32c0393b-145c-472b-95cd-267c8bc7457b-kube-api-access-6c4nt\") pod \"volume-data-source-validator-5d456fdbcd-qb2c9\" (UID: \"32c0393b-145c-472b-95cd-267c8bc7457b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qb2c9"
Jul 08 06:55:20.701219 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.701191    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c8c549564-96gw6"]
Jul 08 06:55:20.704527 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.704502    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-nwhbt"]
Jul 08 06:55:20.704670 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.704652    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.707491 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.707469    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:20.709459 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.709437    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 08 06:55:20.709920 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.709901    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 08 06:55:20.710681 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.710663    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-q8mxm\""
Jul 08 06:55:20.711307 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.711220    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 08 06:55:20.713464 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.713447    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xqsv7\""
Jul 08 06:55:20.713464 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.713454    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 08 06:55:20.713792 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.713774    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 08 06:55:20.715769 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.715738    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 08 06:55:20.716089 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.716054    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 08 06:55:20.728189 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.728147    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-nwhbt"]
Jul 08 06:55:20.730112 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.730083    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 08 06:55:20.730806 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.730787    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 08 06:55:20.732649 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.732633    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c8c549564-96gw6"]
Jul 08 06:55:20.794948 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.794918    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0d34567-03c2-42ac-a6a0-61c6531c483e-config\") pod \"console-operator-68b757fb67-nwhbt\" (UID: \"c0d34567-03c2-42ac-a6a0-61c6531c483e\") " pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:20.795107 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.794964    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-bound-sa-token\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.795107 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.795003    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40d2fe78-b332-4a62-a021-ab10f43a906f-ca-trust-extracted\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.795107 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.795023    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0d34567-03c2-42ac-a6a0-61c6531c483e-serving-cert\") pod \"console-operator-68b757fb67-nwhbt\" (UID: \"c0d34567-03c2-42ac-a6a0-61c6531c483e\") " pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:20.795107 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.795069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0d34567-03c2-42ac-a6a0-61c6531c483e-trusted-ca\") pod \"console-operator-68b757fb67-nwhbt\" (UID: \"c0d34567-03c2-42ac-a6a0-61c6531c483e\") " pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:20.795107 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.795092    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4pxn\" (UniqueName: \"kubernetes.io/projected/c0d34567-03c2-42ac-a6a0-61c6531c483e-kube-api-access-j4pxn\") pod \"console-operator-68b757fb67-nwhbt\" (UID: \"c0d34567-03c2-42ac-a6a0-61c6531c483e\") " pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:20.795364 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.795120    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.795364 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.795214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4nt\" (UniqueName: \"kubernetes.io/projected/32c0393b-145c-472b-95cd-267c8bc7457b-kube-api-access-6c4nt\") pod \"volume-data-source-validator-5d456fdbcd-qb2c9\" (UID: \"32c0393b-145c-472b-95cd-267c8bc7457b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qb2c9"
Jul 08 06:55:20.795364 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.795252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40d2fe78-b332-4a62-a021-ab10f43a906f-image-registry-private-configuration\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.795364 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.795284    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwnk8\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-kube-api-access-hwnk8\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.795364 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.795317    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-certificates\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.795364 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.795342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40d2fe78-b332-4a62-a021-ab10f43a906f-trusted-ca\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.795364 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.795361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40d2fe78-b332-4a62-a021-ab10f43a906f-installation-pull-secrets\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.811996 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.811974    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4nt\" (UniqueName: \"kubernetes.io/projected/32c0393b-145c-472b-95cd-267c8bc7457b-kube-api-access-6c4nt\") pod \"volume-data-source-validator-5d456fdbcd-qb2c9\" (UID: \"32c0393b-145c-472b-95cd-267c8bc7457b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qb2c9"
Jul 08 06:55:20.896683 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.896612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40d2fe78-b332-4a62-a021-ab10f43a906f-image-registry-private-configuration\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.896683 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.896657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwnk8\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-kube-api-access-hwnk8\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.896866 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.896839    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-certificates\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.896922 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.896878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40d2fe78-b332-4a62-a021-ab10f43a906f-trusted-ca\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.897077 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.897052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40d2fe78-b332-4a62-a021-ab10f43a906f-installation-pull-secrets\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.897192 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.897125    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0d34567-03c2-42ac-a6a0-61c6531c483e-config\") pod \"console-operator-68b757fb67-nwhbt\" (UID: \"c0d34567-03c2-42ac-a6a0-61c6531c483e\") " pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:20.897192 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.897186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-bound-sa-token\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.897328 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.897212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40d2fe78-b332-4a62-a021-ab10f43a906f-ca-trust-extracted\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.897328 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.897235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0d34567-03c2-42ac-a6a0-61c6531c483e-serving-cert\") pod \"console-operator-68b757fb67-nwhbt\" (UID: \"c0d34567-03c2-42ac-a6a0-61c6531c483e\") " pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:20.897328 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.897287    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0d34567-03c2-42ac-a6a0-61c6531c483e-trusted-ca\") pod \"console-operator-68b757fb67-nwhbt\" (UID: \"c0d34567-03c2-42ac-a6a0-61c6531c483e\") " pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:20.897328 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.897311    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4pxn\" (UniqueName: \"kubernetes.io/projected/c0d34567-03c2-42ac-a6a0-61c6531c483e-kube-api-access-j4pxn\") pod \"console-operator-68b757fb67-nwhbt\" (UID: \"c0d34567-03c2-42ac-a6a0-61c6531c483e\") " pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:20.897535 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.897342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.897610 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.897579    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-certificates\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.898052 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.897923    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40d2fe78-b332-4a62-a021-ab10f43a906f-ca-trust-extracted\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.898688 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.898358    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0d34567-03c2-42ac-a6a0-61c6531c483e-config\") pod \"console-operator-68b757fb67-nwhbt\" (UID: \"c0d34567-03c2-42ac-a6a0-61c6531c483e\") " pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:20.898688 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.898576    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0d34567-03c2-42ac-a6a0-61c6531c483e-trusted-ca\") pod \"console-operator-68b757fb67-nwhbt\" (UID: \"c0d34567-03c2-42ac-a6a0-61c6531c483e\") " pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:20.898840 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.898707    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40d2fe78-b332-4a62-a021-ab10f43a906f-trusted-ca\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.900533 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:20.900165    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 06:55:20.900533 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:20.900188    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8c549564-96gw6: secret "image-registry-tls" not found
Jul 08 06:55:20.900533 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:20.900253    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls podName:40d2fe78-b332-4a62-a021-ab10f43a906f nodeName:}" failed. No retries permitted until 2026-07-08 06:55:21.40023515 +0000 UTC m=+37.204021532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls") pod "image-registry-6c8c549564-96gw6" (UID: "40d2fe78-b332-4a62-a021-ab10f43a906f") : secret "image-registry-tls" not found
Jul 08 06:55:20.900954 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.900923    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40d2fe78-b332-4a62-a021-ab10f43a906f-installation-pull-secrets\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.902520 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.902496    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40d2fe78-b332-4a62-a021-ab10f43a906f-image-registry-private-configuration\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.902992 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.902955    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0d34567-03c2-42ac-a6a0-61c6531c483e-serving-cert\") pod \"console-operator-68b757fb67-nwhbt\" (UID: \"c0d34567-03c2-42ac-a6a0-61c6531c483e\") " pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:20.906740 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.906700    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-bound-sa-token\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.907275 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.907076    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4pxn\" (UniqueName: \"kubernetes.io/projected/c0d34567-03c2-42ac-a6a0-61c6531c483e-kube-api-access-j4pxn\") pod \"console-operator-68b757fb67-nwhbt\" (UID: \"c0d34567-03c2-42ac-a6a0-61c6531c483e\") " pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:20.907950 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.907517    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwnk8\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-kube-api-access-hwnk8\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:20.911766 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.911742    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qb2c9"
Jul 08 06:55:20.928659 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.928638    2582 generic.go:356] "Generic (PLEG): container finished" podID="1fe61b54-f929-432e-a735-6784ec370581" containerID="3020f8bb511daec7b5d78be739f769e434f64b6ec23bb4e10a1cb6ecca8258bf" exitCode=0
Jul 08 06:55:20.928766 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.928686    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-242sl" event={"ID":"1fe61b54-f929-432e-a735-6784ec370581","Type":"ContainerDied","Data":"3020f8bb511daec7b5d78be739f769e434f64b6ec23bb4e10a1cb6ecca8258bf"}
Jul 08 06:55:20.962467 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:20.962423    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:21.023837 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:21.023638    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:21.071655 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:21.071582    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:21.154281 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:21.154226    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qb2c9"]
Jul 08 06:55:21.157099 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:55:21.157075    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c0393b_145c_472b_95cd_267c8bc7457b.slice/crio-23b1a4340395b22dc868ea934963d06b56fd9c5c87619df4d9d934a3c54e75be WatchSource:0}: Error finding container 23b1a4340395b22dc868ea934963d06b56fd9c5c87619df4d9d934a3c54e75be: Status 404 returned error can't find the container with id 23b1a4340395b22dc868ea934963d06b56fd9c5c87619df4d9d934a3c54e75be
Jul 08 06:55:21.193296 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:21.193275    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-nwhbt"]
Jul 08 06:55:21.196202 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:55:21.196170    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d34567_03c2_42ac_a6a0_61c6531c483e.slice/crio-2575f85ede0fdfbf5bd352bb181b1c8f875757e6a9ed5b7c005396cc93de57b3 WatchSource:0}: Error finding container 2575f85ede0fdfbf5bd352bb181b1c8f875757e6a9ed5b7c005396cc93de57b3: Status 404 returned error can't find the container with id 2575f85ede0fdfbf5bd352bb181b1c8f875757e6a9ed5b7c005396cc93de57b3
Jul 08 06:55:21.400765 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:21.400729    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:21.400917 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:21.400877    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 06:55:21.400917 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:21.400891    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8c549564-96gw6: secret "image-registry-tls" not found
Jul 08 06:55:21.400980 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:21.400944    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls podName:40d2fe78-b332-4a62-a021-ab10f43a906f nodeName:}" failed. No retries permitted until 2026-07-08 06:55:22.400926899 +0000 UTC m=+38.204713296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls") pod "image-registry-6c8c549564-96gw6" (UID: "40d2fe78-b332-4a62-a021-ab10f43a906f") : secret "image-registry-tls" not found
Jul 08 06:55:21.932557 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:21.932521    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wxcjq" event={"ID":"bc6bc5da-ac3a-43be-a3b9-7468831da20e","Type":"ContainerStarted","Data":"6287c1dded0ccd957b239a605b44d68da43b8f5fb7b1c4f231f2abde753fe5c6"}
Jul 08 06:55:21.933863 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:21.933811    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-nwhbt" event={"ID":"c0d34567-03c2-42ac-a6a0-61c6531c483e","Type":"ContainerStarted","Data":"2575f85ede0fdfbf5bd352bb181b1c8f875757e6a9ed5b7c005396cc93de57b3"}
Jul 08 06:55:21.935483 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:21.935461    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qb2c9" event={"ID":"32c0393b-145c-472b-95cd-267c8bc7457b","Type":"ContainerStarted","Data":"23b1a4340395b22dc868ea934963d06b56fd9c5c87619df4d9d934a3c54e75be"}
Jul 08 06:55:21.939457 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:21.939434    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-242sl" event={"ID":"1fe61b54-f929-432e-a735-6784ec370581","Type":"ContainerStarted","Data":"32ca57c4f1992e4c010b1de128bf95926164e638f99257278dd5d86fb1117621"}
Jul 08 06:55:21.976513 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:21.976458    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wxcjq" podStartSLOduration=1.944045631 podStartE2EDuration="3.976439106s" totalImagesPullingTime="2.032393475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:21.975666301 +0000 UTC m=+37.779452721" watchObservedRunningTime="2026-07-08 06:55:21.976439106 +0000 UTC m=+37.780225504"
Jul 08 06:55:22.031089 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:22.031029    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-242sl" podStartSLOduration=10.303077763 podStartE2EDuration="37.031008975s" totalImagesPullingTime="26.727931212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:22.026713382 +0000 UTC m=+37.830499793" watchObservedRunningTime="2026-07-08 06:55:22.031008975 +0000 UTC m=+37.834795373"
Jul 08 06:55:22.108673 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:22.108643    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:22.207237 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:22.207120    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:22.308470 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:22.308439    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:22.308617 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:22.308587    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:55:22.308669 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:22.308649    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls podName:e142a9e3-7af5-4e3a-9b0e-070f68044486 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:26.30863347 +0000 UTC m=+42.112419848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls") pod "dns-default-gjw5t" (UID: "e142a9e3-7af5-4e3a-9b0e-070f68044486") : secret "dns-default-metrics-tls" not found
Jul 08 06:55:22.312043 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:22.312010    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:22.408295 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:22.408257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:22.409314 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:22.409293    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:22.409436 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:22.409424    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 06:55:22.409436 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:22.409437    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8c549564-96gw6: secret "image-registry-tls" not found
Jul 08 06:55:22.409510 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:22.409484    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls podName:40d2fe78-b332-4a62-a021-ab10f43a906f nodeName:}" failed. No retries permitted until 2026-07-08 06:55:24.409470684 +0000 UTC m=+40.213257063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls") pod "image-registry-6c8c549564-96gw6" (UID: "40d2fe78-b332-4a62-a021-ab10f43a906f") : secret "image-registry-tls" not found
Jul 08 06:55:22.944269 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:22.944232    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qb2c9" event={"ID":"32c0393b-145c-472b-95cd-267c8bc7457b","Type":"ContainerStarted","Data":"468b475418efd7dba34521328200467d2210de6328e06e6321a3747b863c04d2"}
Jul 08 06:55:22.962436 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:22.962389    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qb2c9" podStartSLOduration=1.694779697 podStartE2EDuration="2.962371087s" totalImagesPullingTime="1.26759139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:22.961454421 +0000 UTC m=+38.765240820" watchObservedRunningTime="2026-07-08 06:55:22.962371087 +0000 UTC m=+38.766157486"
Jul 08 06:55:23.077345 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:23.077316    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:23.778286 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:23.778221    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:23.841358 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:23.841336    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:23.947555 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:23.947535    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/0.log"
Jul 08 06:55:23.947843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:23.947572    2582 generic.go:356] "Generic (PLEG): container finished" podID="c0d34567-03c2-42ac-a6a0-61c6531c483e" containerID="0b166c3e6729d098273f5b21d18041ba9f8121f3bd1369a2d0afa4b30c29e0d1" exitCode=255
Jul 08 06:55:23.947843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:23.947601    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-nwhbt" event={"ID":"c0d34567-03c2-42ac-a6a0-61c6531c483e","Type":"ContainerDied","Data":"0b166c3e6729d098273f5b21d18041ba9f8121f3bd1369a2d0afa4b30c29e0d1"}
Jul 08 06:55:23.947843 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:23.947825    2582 scope.go:117] "RemoveContainer" containerID="0b166c3e6729d098273f5b21d18041ba9f8121f3bd1369a2d0afa4b30c29e0d1"
Jul 08 06:55:24.426285 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.426254    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:24.426441 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:24.426420    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 06:55:24.426484 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:24.426446    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8c549564-96gw6: secret "image-registry-tls" not found
Jul 08 06:55:24.426538 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:24.426527    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls podName:40d2fe78-b332-4a62-a021-ab10f43a906f nodeName:}" failed. No retries permitted until 2026-07-08 06:55:28.426506429 +0000 UTC m=+44.230292827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls") pod "image-registry-6c8c549564-96gw6" (UID: "40d2fe78-b332-4a62-a021-ab10f43a906f") : secret "image-registry-tls" not found
Jul 08 06:55:24.525354 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.525326    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:24.609562 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.609537    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:24.712919 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.712841    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jvvnp"]
Jul 08 06:55:24.734806 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.734783    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:24.737569 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.737550    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 08 06:55:24.738453 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.738342    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 08 06:55:24.741345 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.738727    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h6mll\""
Jul 08 06:55:24.741345 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.739257    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 08 06:55:24.741493 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.741411    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 08 06:55:24.754470 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.754449    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jvvnp"]
Jul 08 06:55:24.760972 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.760953    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:24.829357 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.829332    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-crio-socket\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:24.829473 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.829391    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:24.829473 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.829412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-data-volume\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:24.829590 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.829509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:24.829590 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.829542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfh8b\" (UniqueName: \"kubernetes.io/projected/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-api-access-pfh8b\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:24.930134 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.930106    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:24.930243 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.930145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-data-volume\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:24.930243 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.930205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:24.930243 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.930231    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfh8b\" (UniqueName: \"kubernetes.io/projected/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-api-access-pfh8b\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:24.930383 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:24.930246    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 06:55:24.930383 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:24.930296    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 06:55:24.930383 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:24.930307    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls podName:3ee5c8ec-83a9-4b40-b2db-25c67997ca4b nodeName:}" failed. No retries permitted until 2026-07-08 06:55:25.430287587 +0000 UTC m=+41.234073974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvvnp" (UID: "3ee5c8ec-83a9-4b40-b2db-25c67997ca4b") : secret "insights-runtime-extractor-tls" not found
Jul 08 06:55:24.930383 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:24.930346    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm podName:3ee5c8ec-83a9-4b40-b2db-25c67997ca4b nodeName:}" failed. No retries permitted until 2026-07-08 06:55:25.430333149 +0000 UTC m=+41.234119554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvvnp" (UID: "3ee5c8ec-83a9-4b40-b2db-25c67997ca4b") : configmap "kube-rbac-proxy" not found
Jul 08 06:55:24.930383 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.930362    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-crio-socket\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:24.930553 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.930506    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-crio-socket\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:24.930553 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.930516    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-data-volume\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:24.941733 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.941712    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfh8b\" (UniqueName: \"kubernetes.io/projected/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-api-access-pfh8b\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:24.951051 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.951034    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 06:55:24.951466 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.951451    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/0.log"
Jul 08 06:55:24.951530 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.951489    2582 generic.go:356] "Generic (PLEG): container finished" podID="c0d34567-03c2-42ac-a6a0-61c6531c483e" containerID="bc1b8692fd97e07aec9a1a3d929c5620730ea81a660f615c1afca870d1c91fa5" exitCode=255
Jul 08 06:55:24.951584 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.951553    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-nwhbt" event={"ID":"c0d34567-03c2-42ac-a6a0-61c6531c483e","Type":"ContainerDied","Data":"bc1b8692fd97e07aec9a1a3d929c5620730ea81a660f615c1afca870d1c91fa5"}
Jul 08 06:55:24.951636 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.951600    2582 scope.go:117] "RemoveContainer" containerID="0b166c3e6729d098273f5b21d18041ba9f8121f3bd1369a2d0afa4b30c29e0d1"
Jul 08 06:55:24.951835 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:24.951816    2582 scope.go:117] "RemoveContainer" containerID="bc1b8692fd97e07aec9a1a3d929c5620730ea81a660f615c1afca870d1c91fa5"
Jul 08 06:55:24.951987 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:24.951962    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-nwhbt_openshift-console-operator(c0d34567-03c2-42ac-a6a0-61c6531c483e)\"" pod="openshift-console-operator/console-operator-68b757fb67-nwhbt" podUID="c0d34567-03c2-42ac-a6a0-61c6531c483e"
Jul 08 06:55:25.433704 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:25.433676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:25.433868 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:25.433755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:25.433868 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:25.433803    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 06:55:25.433868 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:25.433831    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 06:55:25.434033 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:25.433872    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm podName:3ee5c8ec-83a9-4b40-b2db-25c67997ca4b nodeName:}" failed. No retries permitted until 2026-07-08 06:55:26.433853906 +0000 UTC m=+42.237640289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvvnp" (UID: "3ee5c8ec-83a9-4b40-b2db-25c67997ca4b") : configmap "kube-rbac-proxy" not found
Jul 08 06:55:25.434033 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:25.433893    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls podName:3ee5c8ec-83a9-4b40-b2db-25c67997ca4b nodeName:}" failed. No retries permitted until 2026-07-08 06:55:26.433883534 +0000 UTC m=+42.237669913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvvnp" (UID: "3ee5c8ec-83a9-4b40-b2db-25c67997ca4b") : secret "insights-runtime-extractor-tls" not found
Jul 08 06:55:25.955020 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:25.954992    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 06:55:25.955396 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:25.955297    2582 scope.go:117] "RemoveContainer" containerID="bc1b8692fd97e07aec9a1a3d929c5620730ea81a660f615c1afca870d1c91fa5"
Jul 08 06:55:25.955437 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:25.955419    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-nwhbt_openshift-console-operator(c0d34567-03c2-42ac-a6a0-61c6531c483e)\"" pod="openshift-console-operator/console-operator-68b757fb67-nwhbt" podUID="c0d34567-03c2-42ac-a6a0-61c6531c483e"
Jul 08 06:55:26.340471 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:26.340393    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:26.340612 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:26.340516    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:55:26.340612 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:26.340575    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls podName:e142a9e3-7af5-4e3a-9b0e-070f68044486 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:34.340560366 +0000 UTC m=+50.144346744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls") pod "dns-default-gjw5t" (UID: "e142a9e3-7af5-4e3a-9b0e-070f68044486") : secret "dns-default-metrics-tls" not found
Jul 08 06:55:26.441651 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:26.441625    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:26.441728 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:26.441664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:26.441773 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:26.441759    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 06:55:26.441818 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:26.441810    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm podName:3ee5c8ec-83a9-4b40-b2db-25c67997ca4b nodeName:}" failed. No retries permitted until 2026-07-08 06:55:28.441798233 +0000 UTC m=+44.245584610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvvnp" (UID: "3ee5c8ec-83a9-4b40-b2db-25c67997ca4b") : configmap "kube-rbac-proxy" not found
Jul 08 06:55:26.441858 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:26.441759    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 06:55:26.441909 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:26.441900    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls podName:3ee5c8ec-83a9-4b40-b2db-25c67997ca4b nodeName:}" failed. No retries permitted until 2026-07-08 06:55:28.44188766 +0000 UTC m=+44.245674043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvvnp" (UID: "3ee5c8ec-83a9-4b40-b2db-25c67997ca4b") : secret "insights-runtime-extractor-tls" not found
Jul 08 06:55:26.823545 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:26.823510    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:27.005065 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:27.005034    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:27.208043 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:27.207993    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:27.237334 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:27.237311    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6v9zs_8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9/dns-node-resolver/0.log"
Jul 08 06:55:27.307117 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:27.307076    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:27.519964 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:27.519884    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:27.619994 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:27.619967    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:28.236891 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:28.236864    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2tswp_5ae16139-e371-414e-982c-221c0223968a/node-ca/0.log"
Jul 08 06:55:28.456051 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:28.456015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:28.456218 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:28.456085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:28.456218 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:28.456116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:28.456218 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:28.456132    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 06:55:28.456218 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:28.456216    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm podName:3ee5c8ec-83a9-4b40-b2db-25c67997ca4b nodeName:}" failed. No retries permitted until 2026-07-08 06:55:32.456200075 +0000 UTC m=+48.259986453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvvnp" (UID: "3ee5c8ec-83a9-4b40-b2db-25c67997ca4b") : configmap "kube-rbac-proxy" not found
Jul 08 06:55:28.456365 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:28.456224    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 06:55:28.456365 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:28.456242    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 06:55:28.456365 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:28.456260    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8c549564-96gw6: secret "image-registry-tls" not found
Jul 08 06:55:28.456365 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:28.456262    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls podName:3ee5c8ec-83a9-4b40-b2db-25c67997ca4b nodeName:}" failed. No retries permitted until 2026-07-08 06:55:32.456251801 +0000 UTC m=+48.260038180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvvnp" (UID: "3ee5c8ec-83a9-4b40-b2db-25c67997ca4b") : secret "insights-runtime-extractor-tls" not found
Jul 08 06:55:28.456365 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:28.456308    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls podName:40d2fe78-b332-4a62-a021-ab10f43a906f nodeName:}" failed. No retries permitted until 2026-07-08 06:55:36.456297136 +0000 UTC m=+52.260083520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls") pod "image-registry-6c8c549564-96gw6" (UID: "40d2fe78-b332-4a62-a021-ab10f43a906f") : secret "image-registry-tls" not found
Jul 08 06:55:29.124645 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.124614    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-7wdtf"]
Jul 08 06:55:29.156415 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.156386    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:29.172232 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.172207    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-7wdtf"]
Jul 08 06:55:29.172319 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.172284    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-7wdtf"
Jul 08 06:55:29.175400 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.175374    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 08 06:55:29.175495 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.175374    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 08 06:55:29.175495 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.175380    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 08 06:55:29.175495 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.175380    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 08 06:55:29.176623 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.176608    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mknll\""
Jul 08 06:55:29.234826 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.234805    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:29.236447 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.236432    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wxcjq_bc6bc5da-ac3a-43be-a3b9-7468831da20e/serve-healthcheck-canary/0.log"
Jul 08 06:55:29.261757 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.261735    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830-signing-key\") pod \"service-ca-758f599c5-7wdtf\" (UID: \"7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830\") " pod="openshift-service-ca/service-ca-758f599c5-7wdtf"
Jul 08 06:55:29.262055 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.261825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc9pm\" (UniqueName: \"kubernetes.io/projected/7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830-kube-api-access-zc9pm\") pod \"service-ca-758f599c5-7wdtf\" (UID: \"7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830\") " pod="openshift-service-ca/service-ca-758f599c5-7wdtf"
Jul 08 06:55:29.262055 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.261852    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830-signing-cabundle\") pod \"service-ca-758f599c5-7wdtf\" (UID: \"7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830\") " pod="openshift-service-ca/service-ca-758f599c5-7wdtf"
Jul 08 06:55:29.362798 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.362778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc9pm\" (UniqueName: \"kubernetes.io/projected/7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830-kube-api-access-zc9pm\") pod \"service-ca-758f599c5-7wdtf\" (UID: \"7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830\") " pod="openshift-service-ca/service-ca-758f599c5-7wdtf"
Jul 08 06:55:29.362869 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.362807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830-signing-cabundle\") pod \"service-ca-758f599c5-7wdtf\" (UID: \"7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830\") " pod="openshift-service-ca/service-ca-758f599c5-7wdtf"
Jul 08 06:55:29.362869 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.362846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830-signing-key\") pod \"service-ca-758f599c5-7wdtf\" (UID: \"7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830\") " pod="openshift-service-ca/service-ca-758f599c5-7wdtf"
Jul 08 06:55:29.363545 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.363524    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830-signing-cabundle\") pod \"service-ca-758f599c5-7wdtf\" (UID: \"7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830\") " pod="openshift-service-ca/service-ca-758f599c5-7wdtf"
Jul 08 06:55:29.365188 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.365171    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830-signing-key\") pod \"service-ca-758f599c5-7wdtf\" (UID: \"7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830\") " pod="openshift-service-ca/service-ca-758f599c5-7wdtf"
Jul 08 06:55:29.375255 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.375205    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc9pm\" (UniqueName: \"kubernetes.io/projected/7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830-kube-api-access-zc9pm\") pod \"service-ca-758f599c5-7wdtf\" (UID: \"7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830\") " pod="openshift-service-ca/service-ca-758f599c5-7wdtf"
Jul 08 06:55:29.480988 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.480964    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-7wdtf"
Jul 08 06:55:29.625448 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.625402    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-7wdtf"]
Jul 08 06:55:29.627771 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:55:29.627726    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa1ac81_cb95_4dc6_82aa_dcfcfdd41830.slice/crio-232c0063107d34d060df08411b244b0337e02782a3e75513cb0525d91c7d1e69 WatchSource:0}: Error finding container 232c0063107d34d060df08411b244b0337e02782a3e75513cb0525d91c7d1e69: Status 404 returned error can't find the container with id 232c0063107d34d060df08411b244b0337e02782a3e75513cb0525d91c7d1e69
Jul 08 06:55:29.964589 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:29.964557    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-7wdtf" event={"ID":"7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830","Type":"ContainerStarted","Data":"232c0063107d34d060df08411b244b0337e02782a3e75513cb0525d91c7d1e69"}
Jul 08 06:55:30.253844 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:30.253759    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:31.024299 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:31.024259    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:31.024784 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:31.024329    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:31.024856 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:31.024785    2582 scope.go:117] "RemoveContainer" containerID="bc1b8692fd97e07aec9a1a3d929c5620730ea81a660f615c1afca870d1c91fa5"
Jul 08 06:55:31.024999 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:31.024975    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-nwhbt_openshift-console-operator(c0d34567-03c2-42ac-a6a0-61c6531c483e)\"" pod="openshift-console-operator/console-operator-68b757fb67-nwhbt" podUID="c0d34567-03c2-42ac-a6a0-61c6531c483e"
Jul 08 06:55:31.971235 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:31.971205    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-7wdtf" event={"ID":"7aa1ac81-cb95-4dc6-82aa-dcfcfdd41830","Type":"ContainerStarted","Data":"b357c54177426898dd796e65219ef1aca5859045bf2763df77c12e0eddca61f5"}
Jul 08 06:55:31.996697 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:31.996609    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-7wdtf" podStartSLOduration=0.896722878 podStartE2EDuration="2.996592264s" totalImagesPullingTime="2.099869386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:31.99132831 +0000 UTC m=+47.795114710" watchObservedRunningTime="2026-07-08 06:55:31.996592264 +0000 UTC m=+47.800378688"
Jul 08 06:55:32.487108 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:32.487078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:32.487480 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:32.487135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:32.487480 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:32.487208    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 06:55:32.487480 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:32.487269    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 06:55:32.487480 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:32.487272    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm podName:3ee5c8ec-83a9-4b40-b2db-25c67997ca4b nodeName:}" failed. No retries permitted until 2026-07-08 06:55:40.487255622 +0000 UTC m=+56.291042017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jvvnp" (UID: "3ee5c8ec-83a9-4b40-b2db-25c67997ca4b") : configmap "kube-rbac-proxy" not found
Jul 08 06:55:32.487480 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:32.487314    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls podName:3ee5c8ec-83a9-4b40-b2db-25c67997ca4b nodeName:}" failed. No retries permitted until 2026-07-08 06:55:40.48730329 +0000 UTC m=+56.291089668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvvnp" (UID: "3ee5c8ec-83a9-4b40-b2db-25c67997ca4b") : secret "insights-runtime-extractor-tls" not found
Jul 08 06:55:33.193877 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:33.193841    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:33.323615 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:33.323581    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:33.476650 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:33.476572    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:33.604503 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:33.604471    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:33.721633 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:33.721602    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:33.829458 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:33.829397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:33.967372 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:33.967343    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:34.389511 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:34.389483    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:34.401548 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:34.401528    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:34.401636 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:34.401623    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:55:34.401686 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:34.401676    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls podName:e142a9e3-7af5-4e3a-9b0e-070f68044486 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:50.401661739 +0000 UTC m=+66.205448119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls") pod "dns-default-gjw5t" (UID: "e142a9e3-7af5-4e3a-9b0e-070f68044486") : secret "dns-default-metrics-tls" not found
Jul 08 06:55:34.898475 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:34.898441    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:35.292825 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:35.292800    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:35.505957 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:35.505925    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:35.707997 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:35.707966    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:35.897193 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:35.897166    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:36.517200 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:36.517166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:36.517666 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:36.517347    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 06:55:36.517666 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:36.517369    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8c549564-96gw6: secret "image-registry-tls" not found
Jul 08 06:55:36.517666 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:36.517443    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls podName:40d2fe78-b332-4a62-a021-ab10f43a906f nodeName:}" failed. No retries permitted until 2026-07-08 06:55:52.517421745 +0000 UTC m=+68.321208143 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls") pod "image-registry-6c8c549564-96gw6" (UID: "40d2fe78-b332-4a62-a021-ab10f43a906f") : secret "image-registry-tls" not found
Jul 08 06:55:36.904360 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:36.904285    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:37.006833 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:37.006810    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:37.298086 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:37.298063    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:37.484579 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:37.484548    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:37.687102 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:37.687072    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:37.761395 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:37.761367    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:37.889864 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:37.889839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:39.177728 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:39.177693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:39.287426 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:39.287386    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:39.449569 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:39.449490    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:39.552246 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:39.552219    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:39.690282 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:39.690255    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:39.887257 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:39.887141    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:40.541895 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:40.541864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:40.542292 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:40.541940    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:40.542292 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:40.542000    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 06:55:40.542292 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:55:40.542054    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls podName:3ee5c8ec-83a9-4b40-b2db-25c67997ca4b nodeName:}" failed. No retries permitted until 2026-07-08 06:55:56.542038358 +0000 UTC m=+72.345824740 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jvvnp" (UID: "3ee5c8ec-83a9-4b40-b2db-25c67997ca4b") : secret "insights-runtime-extractor-tls" not found
Jul 08 06:55:40.542407 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:40.542395    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:41.088631 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:41.088600    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:41.295865 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:41.295827    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:41.482695 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:41.482661    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:41.579924 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:41.579891    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:41.691284 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:41.691251    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:41.894876 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:41.894799    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:42.888397 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:42.888364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:42.908921 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:42.908892    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzsh8"
Jul 08 06:55:43.090677 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:43.090649    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:43.292987 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:43.292955    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:43.490788 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:43.490758    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:43.934096 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:43.934069    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:45.740984 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:45.740949    2582 scope.go:117] "RemoveContainer" containerID="bc1b8692fd97e07aec9a1a3d929c5620730ea81a660f615c1afca870d1c91fa5"
Jul 08 06:55:46.007422 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:46.007354    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 06:55:46.007422 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:46.007404    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-nwhbt" event={"ID":"c0d34567-03c2-42ac-a6a0-61c6531c483e","Type":"ContainerStarted","Data":"dd2885f9697435af4dd3962a9dc46a4ef54d6004193682816093e2eea54bbece"}
Jul 08 06:55:46.007693 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:46.007674    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:46.027567 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:46.027500    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-nwhbt" podStartSLOduration=23.774069842 podStartE2EDuration="26.027487505s" totalImagesPullingTime="2.253417663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:46.026445478 +0000 UTC m=+61.830231880" watchObservedRunningTime="2026-07-08 06:55:46.027487505 +0000 UTC m=+61.831273943"
Jul 08 06:55:46.354739 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:46.354660    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-nwhbt"
Jul 08 06:55:46.559063 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:46.559028    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:46.669930 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:46.669901    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:46.840987 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:46.840962    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:47.222454 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:47.222426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:47.324359 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:47.324317    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:47.422797 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:47.422771    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:47.488583 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:47.488530    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:47.609608 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:47.609585    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:47.672714 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:47.672689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:48.240593 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:48.240565    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:48.421903 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:48.421877    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:48.580477 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:48.580407    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:48.843787 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:48.843707    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:48.919344 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:48.919321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:50.418922 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.418892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:50.421143 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.421124    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e142a9e3-7af5-4e3a-9b0e-070f68044486-metrics-tls\") pod \"dns-default-gjw5t\" (UID: \"e142a9e3-7af5-4e3a-9b0e-070f68044486\") " pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:50.519371 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.519342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:50.519473 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.519381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs\") pod \"network-metrics-daemon-h7cff\" (UID: \"d397da7e-ef4a-4e32-a8f3-3888459f52b3\") " pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:50.522225 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.522207    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 06:55:50.522301 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.522270    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 06:55:50.532370 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.532349    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d397da7e-ef4a-4e32-a8f3-3888459f52b3-metrics-certs\") pod \"network-metrics-daemon-h7cff\" (UID: \"d397da7e-ef4a-4e32-a8f3-3888459f52b3\") " pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:50.532495 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.532474    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf315f9-c635-4d76-81fb-39beef3d0746-original-pull-secret\") pod \"global-pull-secret-syncer-9snpb\" (UID: \"dbf315f9-c635-4d76-81fb-39beef3d0746\") " pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:50.619587 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.619561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlb6\" (UniqueName: \"kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6\") pod \"network-check-target-qq8lz\" (UID: \"ab22f5b5-5417-4d54-aec8-f5bb631ecd37\") " pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:50.622384 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.622367    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 06:55:50.633339 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.633320    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 06:55:50.643651 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.643624    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdlb6\" (UniqueName: \"kubernetes.io/projected/ab22f5b5-5417-4d54-aec8-f5bb631ecd37-kube-api-access-xdlb6\") pod \"network-check-target-qq8lz\" (UID: \"ab22f5b5-5417-4d54-aec8-f5bb631ecd37\") " pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:50.652487 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.652468    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9snpb"
Jul 08 06:55:50.661281 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.661265    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4f688\""
Jul 08 06:55:50.669636 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.669595    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7cff"
Jul 08 06:55:50.677624 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.677602    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6zr5\""
Jul 08 06:55:50.685700 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.685684    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:50.826340 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.826313    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9snpb"]
Jul 08 06:55:50.829617 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:55:50.829588    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf315f9_c635_4d76_81fb_39beef3d0746.slice/crio-cdc63b911f25203f750e145b95b41e1217d67aa0ffdc0f7eee2b71a5bb12ba84 WatchSource:0}: Error finding container cdc63b911f25203f750e145b95b41e1217d67aa0ffdc0f7eee2b71a5bb12ba84: Status 404 returned error can't find the container with id cdc63b911f25203f750e145b95b41e1217d67aa0ffdc0f7eee2b71a5bb12ba84
Jul 08 06:55:50.844012 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.843992    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h7cff"]
Jul 08 06:55:50.846378 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:55:50.846354    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd397da7e_ef4a_4e32_a8f3_3888459f52b3.slice/crio-bdbd56b365fb7eab4113146e1ece21fd56b29d800b31fad10a579532196902c8 WatchSource:0}: Error finding container bdbd56b365fb7eab4113146e1ece21fd56b29d800b31fad10a579532196902c8: Status 404 returned error can't find the container with id bdbd56b365fb7eab4113146e1ece21fd56b29d800b31fad10a579532196902c8
Jul 08 06:55:50.851213 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.851194    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6n2tj\""
Jul 08 06:55:50.859891 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.859858    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:50.868541 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:50.868521    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gjw5t"]
Jul 08 06:55:50.879884 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:55:50.879865    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode142a9e3_7af5_4e3a_9b0e_070f68044486.slice/crio-7b2540cd5445bd4890956c01fe80da5d43f393a41c892ce64ebeced598382a63 WatchSource:0}: Error finding container 7b2540cd5445bd4890956c01fe80da5d43f393a41c892ce64ebeced598382a63: Status 404 returned error can't find the container with id 7b2540cd5445bd4890956c01fe80da5d43f393a41c892ce64ebeced598382a63
Jul 08 06:55:51.009891 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:51.009869    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qq8lz"]
Jul 08 06:55:51.011933 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:55:51.011904    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab22f5b5_5417_4d54_aec8_f5bb631ecd37.slice/crio-3aa486ec7071ee69aa9fdf0b1929674e10622923191f5d0cabca8b3ec4d9d8e8 WatchSource:0}: Error finding container 3aa486ec7071ee69aa9fdf0b1929674e10622923191f5d0cabca8b3ec4d9d8e8: Status 404 returned error can't find the container with id 3aa486ec7071ee69aa9fdf0b1929674e10622923191f5d0cabca8b3ec4d9d8e8
Jul 08 06:55:51.020189 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:51.020163    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gjw5t" event={"ID":"e142a9e3-7af5-4e3a-9b0e-070f68044486","Type":"ContainerStarted","Data":"7b2540cd5445bd4890956c01fe80da5d43f393a41c892ce64ebeced598382a63"}
Jul 08 06:55:51.021224 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:51.021198    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qq8lz" event={"ID":"ab22f5b5-5417-4d54-aec8-f5bb631ecd37","Type":"ContainerStarted","Data":"3aa486ec7071ee69aa9fdf0b1929674e10622923191f5d0cabca8b3ec4d9d8e8"}
Jul 08 06:55:51.022165 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:51.022133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7cff" event={"ID":"d397da7e-ef4a-4e32-a8f3-3888459f52b3","Type":"ContainerStarted","Data":"bdbd56b365fb7eab4113146e1ece21fd56b29d800b31fad10a579532196902c8"}
Jul 08 06:55:51.023075 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:51.023059    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9snpb" event={"ID":"dbf315f9-c635-4d76-81fb-39beef3d0746","Type":"ContainerStarted","Data":"cdc63b911f25203f750e145b95b41e1217d67aa0ffdc0f7eee2b71a5bb12ba84"}
Jul 08 06:55:52.536122 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:52.536073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:52.539129 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:52.539074    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls\") pod \"image-registry-6c8c549564-96gw6\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") " pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:52.819581 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:52.819384    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-q8mxm\""
Jul 08 06:55:52.827702 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:52.827677    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:53.122047 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.121983    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"]
Jul 08 06:55:53.138225 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.138199    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"]
Jul 08 06:55:53.138351 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.138264    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"
Jul 08 06:55:53.141720 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.141697    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 08 06:55:53.142477 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.142332    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 08 06:55:53.142949 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.142928    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 08 06:55:53.143168 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.143140    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 08 06:55:53.156337 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.156309    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"]
Jul 08 06:55:53.156455 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.156353    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"]
Jul 08 06:55:53.156455 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.156384    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.160228 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.159643    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 08 06:55:53.160228 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.159819    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 08 06:55:53.160228 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.159849    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 08 06:55:53.160228 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.160061    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 08 06:55:53.242425 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.242352    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee68ac5a-bbdf-49a6-96d5-1172dec57398-tmp\") pod \"klusterlet-addon-workmgr-78684896b6-xbsjq\" (UID: \"ee68ac5a-bbdf-49a6-96d5-1172dec57398\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"
Jul 08 06:55:53.242425 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.242397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvk5x\" (UniqueName: \"kubernetes.io/projected/88108892-3043-4b70-a0ea-c9ae85f8396c-kube-api-access-rvk5x\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.242624 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.242471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/88108892-3043-4b70-a0ea-c9ae85f8396c-ca\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.242624 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.242561    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/88108892-3043-4b70-a0ea-c9ae85f8396c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.242624 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.242597    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ee68ac5a-bbdf-49a6-96d5-1172dec57398-klusterlet-config\") pod \"klusterlet-addon-workmgr-78684896b6-xbsjq\" (UID: \"ee68ac5a-bbdf-49a6-96d5-1172dec57398\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"
Jul 08 06:55:53.242775 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.242642    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88108892-3043-4b70-a0ea-c9ae85f8396c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.242775 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.242668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/88108892-3043-4b70-a0ea-c9ae85f8396c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.242775 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.242694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spc7w\" (UniqueName: \"kubernetes.io/projected/ee68ac5a-bbdf-49a6-96d5-1172dec57398-kube-api-access-spc7w\") pod \"klusterlet-addon-workmgr-78684896b6-xbsjq\" (UID: \"ee68ac5a-bbdf-49a6-96d5-1172dec57398\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"
Jul 08 06:55:53.242775 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.242729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/88108892-3043-4b70-a0ea-c9ae85f8396c-hub\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.245921 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.245869    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c8c549564-96gw6"]
Jul 08 06:55:53.263420 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.263394    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:53.345574 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.345317    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/88108892-3043-4b70-a0ea-c9ae85f8396c-hub\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.345574 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.345378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee68ac5a-bbdf-49a6-96d5-1172dec57398-tmp\") pod \"klusterlet-addon-workmgr-78684896b6-xbsjq\" (UID: \"ee68ac5a-bbdf-49a6-96d5-1172dec57398\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"
Jul 08 06:55:53.345574 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.345415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvk5x\" (UniqueName: \"kubernetes.io/projected/88108892-3043-4b70-a0ea-c9ae85f8396c-kube-api-access-rvk5x\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.345574 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.345442    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/88108892-3043-4b70-a0ea-c9ae85f8396c-ca\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.347430 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.346284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/88108892-3043-4b70-a0ea-c9ae85f8396c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.347430 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.346315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ee68ac5a-bbdf-49a6-96d5-1172dec57398-klusterlet-config\") pod \"klusterlet-addon-workmgr-78684896b6-xbsjq\" (UID: \"ee68ac5a-bbdf-49a6-96d5-1172dec57398\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"
Jul 08 06:55:53.347430 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.346374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88108892-3043-4b70-a0ea-c9ae85f8396c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.347430 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.346405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/88108892-3043-4b70-a0ea-c9ae85f8396c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.347430 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.346433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spc7w\" (UniqueName: \"kubernetes.io/projected/ee68ac5a-bbdf-49a6-96d5-1172dec57398-kube-api-access-spc7w\") pod \"klusterlet-addon-workmgr-78684896b6-xbsjq\" (UID: \"ee68ac5a-bbdf-49a6-96d5-1172dec57398\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"
Jul 08 06:55:53.347430 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.347320    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee68ac5a-bbdf-49a6-96d5-1172dec57398-tmp\") pod \"klusterlet-addon-workmgr-78684896b6-xbsjq\" (UID: \"ee68ac5a-bbdf-49a6-96d5-1172dec57398\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"
Jul 08 06:55:53.347796 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.347530    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/88108892-3043-4b70-a0ea-c9ae85f8396c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.351229 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.351204    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/88108892-3043-4b70-a0ea-c9ae85f8396c-hub\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.351360 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.351335    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ee68ac5a-bbdf-49a6-96d5-1172dec57398-klusterlet-config\") pod \"klusterlet-addon-workmgr-78684896b6-xbsjq\" (UID: \"ee68ac5a-bbdf-49a6-96d5-1172dec57398\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"
Jul 08 06:55:53.352383 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.352359    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/88108892-3043-4b70-a0ea-c9ae85f8396c-ca\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.353249 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.353213    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88108892-3043-4b70-a0ea-c9ae85f8396c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.354334 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.354308    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/88108892-3043-4b70-a0ea-c9ae85f8396c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.372745 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.372489    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spc7w\" (UniqueName: \"kubernetes.io/projected/ee68ac5a-bbdf-49a6-96d5-1172dec57398-kube-api-access-spc7w\") pod \"klusterlet-addon-workmgr-78684896b6-xbsjq\" (UID: \"ee68ac5a-bbdf-49a6-96d5-1172dec57398\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"
Jul 08 06:55:53.373662 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.373426    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvk5x\" (UniqueName: \"kubernetes.io/projected/88108892-3043-4b70-a0ea-c9ae85f8396c-kube-api-access-rvk5x\") pod \"cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r\" (UID: \"88108892-3043-4b70-a0ea-c9ae85f8396c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.384110 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.384084    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:53.403346 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.403320    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c8c549564-96gw6"]
Jul 08 06:55:53.407330 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:55:53.407294    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d2fe78_b332_4a62_a021_ab10f43a906f.slice/crio-25f22db1d22e816a7dc67cbe0b67e0276c4d16071b6154aa1b2b0faaf1ab58f4 WatchSource:0}: Error finding container 25f22db1d22e816a7dc67cbe0b67e0276c4d16071b6154aa1b2b0faaf1ab58f4: Status 404 returned error can't find the container with id 25f22db1d22e816a7dc67cbe0b67e0276c4d16071b6154aa1b2b0faaf1ab58f4
Jul 08 06:55:53.451953 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.451719    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"
Jul 08 06:55:53.480959 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.480801    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"
Jul 08 06:55:53.483598 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.483550    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:53.593639 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.593254    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:53.668645 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.668186    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"]
Jul 08 06:55:53.678291 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:55:53.678259    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee68ac5a_bbdf_49a6_96d5_1172dec57398.slice/crio-6f473cab5eb2b9c669b40cf7b8126c771b8683bdb1b3cc4bfb9fab41bcde5a67 WatchSource:0}: Error finding container 6f473cab5eb2b9c669b40cf7b8126c771b8683bdb1b3cc4bfb9fab41bcde5a67: Status 404 returned error can't find the container with id 6f473cab5eb2b9c669b40cf7b8126c771b8683bdb1b3cc4bfb9fab41bcde5a67
Jul 08 06:55:53.714710 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.714645    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r"]
Jul 08 06:55:53.720228 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:53.720128    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:54.037209 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.037168    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7cff" event={"ID":"d397da7e-ef4a-4e32-a8f3-3888459f52b3","Type":"ContainerStarted","Data":"e941f852c8da1d04ebb0e824929fcadeaef67a2f9a1ad038d50183fcc3071d7d"}
Jul 08 06:55:54.037433 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.037235    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7cff" event={"ID":"d397da7e-ef4a-4e32-a8f3-3888459f52b3","Type":"ContainerStarted","Data":"69f40309b4b894b3472dabb5f201b09848610b916158b3c5d5fcede3b609a79e"}
Jul 08 06:55:54.038588 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.038560    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq" event={"ID":"ee68ac5a-bbdf-49a6-96d5-1172dec57398","Type":"ContainerStarted","Data":"6f473cab5eb2b9c669b40cf7b8126c771b8683bdb1b3cc4bfb9fab41bcde5a67"}
Jul 08 06:55:54.040192 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.040171    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c8c549564-96gw6" event={"ID":"40d2fe78-b332-4a62-a021-ab10f43a906f","Type":"ContainerStarted","Data":"de7ab350d4a6a4c7a1217b8cccef37592956ac4762c2490c2627e20a84cf3d06"}
Jul 08 06:55:54.040295 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.040204    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c8c549564-96gw6" event={"ID":"40d2fe78-b332-4a62-a021-ab10f43a906f","Type":"ContainerStarted","Data":"25f22db1d22e816a7dc67cbe0b67e0276c4d16071b6154aa1b2b0faaf1ab58f4"}
Jul 08 06:55:54.040295 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.040252    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:55:54.042205 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.042179    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gjw5t" event={"ID":"e142a9e3-7af5-4e3a-9b0e-070f68044486","Type":"ContainerStarted","Data":"cafe44bcf7ba0fdcf81b9d95c9942b665c3f7a3ed4d5045f7207a42079c23575"}
Jul 08 06:55:54.042307 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.042226    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gjw5t" event={"ID":"e142a9e3-7af5-4e3a-9b0e-070f68044486","Type":"ContainerStarted","Data":"2d775a2f881076552f2b93d9e5ebef00d579a42b65a0802fff24ded410b0a7de"}
Jul 08 06:55:54.042379 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.042332    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:55:54.069100 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.069015    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h7cff" podStartSLOduration=66.69845506 podStartE2EDuration="1m9.068998028s" totalImagesPullingTime="2.370542968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:54.067475112 +0000 UTC m=+69.871261525" watchObservedRunningTime="2026-07-08 06:55:54.068998028 +0000 UTC m=+69.872784427"
Jul 08 06:55:54.115648 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.115598    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gjw5t" podStartSLOduration=33.768719694 podStartE2EDuration="36.115577973s" totalImagesPullingTime="2.346858279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:54.114634788 +0000 UTC m=+69.918421188" watchObservedRunningTime="2026-07-08 06:55:54.115577973 +0000 UTC m=+69.919364373"
Jul 08 06:55:54.139642 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.139592    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c8c549564-96gw6" podStartSLOduration=34.139573044 podStartE2EDuration="34.139573044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:54.139036718 +0000 UTC m=+69.942823139" watchObservedRunningTime="2026-07-08 06:55:54.139573044 +0000 UTC m=+69.943359449"
Jul 08 06:55:54.555922 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.555872    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:54.679072 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:55:54.679040    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88108892_3043_4b70_a0ea_c9ae85f8396c.slice/crio-476385f1b422f30e1278f61df6cf58c4d87369c6cde8d8b3b8ab4123bb3f79e1 WatchSource:0}: Error finding container 476385f1b422f30e1278f61df6cf58c4d87369c6cde8d8b3b8ab4123bb3f79e1: Status 404 returned error can't find the container with id 476385f1b422f30e1278f61df6cf58c4d87369c6cde8d8b3b8ab4123bb3f79e1
Jul 08 06:55:54.856346 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.856213    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:54.978171 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:54.978120    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:55.048396 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:55.048354    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r" event={"ID":"88108892-3043-4b70-a0ea-c9ae85f8396c","Type":"ContainerStarted","Data":"476385f1b422f30e1278f61df6cf58c4d87369c6cde8d8b3b8ab4123bb3f79e1"}
Jul 08 06:55:55.095726 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:55.095689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:55.194525 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:55.194495    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:56.572457 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:56.571841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:56.590252 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:56.590222    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ee5c8ec-83a9-4b40-b2db-25c67997ca4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jvvnp\" (UID: \"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b\") " pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:56.853421 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:56.853119    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h6mll\""
Jul 08 06:55:56.859639 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:56.859394    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jvvnp"
Jul 08 06:55:57.040998 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:57.040944    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jvvnp"]
Jul 08 06:55:57.045850 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:55:57.045813    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee5c8ec_83a9_4b40_b2db_25c67997ca4b.slice/crio-8e02181142c15948ff3cc200b4874511153101b688ff6fbac1eaf26ffb00312a WatchSource:0}: Error finding container 8e02181142c15948ff3cc200b4874511153101b688ff6fbac1eaf26ffb00312a: Status 404 returned error can't find the container with id 8e02181142c15948ff3cc200b4874511153101b688ff6fbac1eaf26ffb00312a
Jul 08 06:55:57.059901 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:57.059325    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qq8lz" event={"ID":"ab22f5b5-5417-4d54-aec8-f5bb631ecd37","Type":"ContainerStarted","Data":"096eeab405a32b68a7ee7c758fdad176c341f4351dfdf97136f7c3d80d96abf0"}
Jul 08 06:55:57.059901 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:57.059672    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:55:57.062039 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:57.061618    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9snpb" event={"ID":"dbf315f9-c635-4d76-81fb-39beef3d0746","Type":"ContainerStarted","Data":"39623bc9666b116ecf752ca7a248a733987d6f445b16e4380a987e6cbfa70600"}
Jul 08 06:55:57.064734 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:57.064659    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvvnp" event={"ID":"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b","Type":"ContainerStarted","Data":"8e02181142c15948ff3cc200b4874511153101b688ff6fbac1eaf26ffb00312a"}
Jul 08 06:55:57.081739 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:57.081040    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qq8lz" podStartSLOduration=66.977730338 podStartE2EDuration="1m12.081020832s" totalImagesPullingTime="5.103290494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:57.079538663 +0000 UTC m=+72.883325065" watchObservedRunningTime="2026-07-08 06:55:57.081020832 +0000 UTC m=+72.884807227"
Jul 08 06:55:57.097004 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:57.095878    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9snpb" podStartSLOduration=66.80452769 podStartE2EDuration="1m12.095864068s" totalImagesPullingTime="5.291336378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:57.095757942 +0000 UTC m=+72.899544343" watchObservedRunningTime="2026-07-08 06:55:57.095864068 +0000 UTC m=+72.899650472"
Jul 08 06:55:57.264490 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:57.264421    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:57.343599 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:57.343517    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:57.476201 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:57.476066    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:57.582413 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:57.582294    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:59.532340 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:59.532310    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:55:59.598779 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:55:59.598742    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:00.112089 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:00.112051    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:00.196369 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:00.196336    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:00.420567 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:00.420534    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:02.272167 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:02.272127    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:02.369409 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:02.369376    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:02.467322 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:02.467296    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:02.538637 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:02.538564    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:03.759693 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:03.759661    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:04.050275 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:04.050252    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gjw5t"
Jul 08 06:56:04.094166 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:04.094127    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq" event={"ID":"ee68ac5a-bbdf-49a6-96d5-1172dec57398","Type":"ContainerStarted","Data":"e5eb952a188ba6374e51d9bb5e4863de2d739882ba98106db298a847b6f1c93b"}
Jul 08 06:56:04.094396 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:04.094376    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"
Jul 08 06:56:04.095920 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:04.095897    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq"
Jul 08 06:56:04.096690 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:04.096667    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvvnp" event={"ID":"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b","Type":"ContainerStarted","Data":"a0260cd367d91067b08eaf3b55a0f7228619c9819b4cfd390462aa9085ff0461"}
Jul 08 06:56:04.096804 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:04.096720    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvvnp" event={"ID":"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b","Type":"ContainerStarted","Data":"a2c327e6f7236d13d543e6b423c3beab72314d5c138d2ad82a82c52c71b5d885"}
Jul 08 06:56:04.098993 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:04.098973    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r" event={"ID":"88108892-3043-4b70-a0ea-c9ae85f8396c","Type":"ContainerStarted","Data":"794fb0e42aad5eb6d4039fd0d8249dd2209e89afdd2fc8fc8b4b40408cabed46"}
Jul 08 06:56:04.099092 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:04.099009    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r" event={"ID":"88108892-3043-4b70-a0ea-c9ae85f8396c","Type":"ContainerStarted","Data":"fef341f59cf03ef515c9aac46294f0a66139aae608ffa231a28339b127e8e0bd"}
Jul 08 06:56:04.099092 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:04.099022    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r" event={"ID":"88108892-3043-4b70-a0ea-c9ae85f8396c","Type":"ContainerStarted","Data":"4fe507d61858a61dd69219c6c48ca688f32025ccd364b8065ecde1e8eff039ee"}
Jul 08 06:56:04.116275 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:04.116232    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78684896b6-xbsjq" podStartSLOduration=1.5517257770000001 podStartE2EDuration="11.116219473s" totalImagesPullingTime="9.564493696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:56:04.115612534 +0000 UTC m=+79.919398958" watchObservedRunningTime="2026-07-08 06:56:04.116219473 +0000 UTC m=+79.920005872"
Jul 08 06:56:04.157810 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:04.157769    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cfdcfc6b4-7jz4r" podStartSLOduration=2.5927493520000002 podStartE2EDuration="11.157754844s" totalImagesPullingTime="8.565005492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:56:04.156528101 +0000 UTC m=+79.960314548" watchObservedRunningTime="2026-07-08 06:56:04.157754844 +0000 UTC m=+79.961541235"
Jul 08 06:56:05.495738 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:05.495686    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:05.676931 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:05.676894    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:06.393645 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.393607    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:06.626263 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.626224    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:06.775938 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.775907    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4wx47"]
Jul 08 06:56:06.779206 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.779185    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.787110 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.786975    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 08 06:56:06.787110 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.787031    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 08 06:56:06.788573 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.788553    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 08 06:56:06.788713 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.788691    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 08 06:56:06.788938 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.788921    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 08 06:56:06.789248 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.789226    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 08 06:56:06.789358 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.789257    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zvxdg\""
Jul 08 06:56:06.849218 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.849192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-accelerators-collector-config\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.849347 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.849245    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-textfile\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.849347 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.849279    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e344bb8-5819-4b55-a499-dea024444e95-root\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.849347 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.849303    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e344bb8-5819-4b55-a499-dea024444e95-metrics-client-ca\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.849347 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.849337    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-tls\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.849558 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.849358    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jzxc\" (UniqueName: \"kubernetes.io/projected/8e344bb8-5819-4b55-a499-dea024444e95-kube-api-access-5jzxc\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.849558 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.849397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e344bb8-5819-4b55-a499-dea024444e95-sys\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.849558 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.849437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.849558 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.849514    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-wtmp\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.877670 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.877632    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:06.950795 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.950776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-wtmp\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.950914 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.950809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-accelerators-collector-config\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.950914 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.950834    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-textfile\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.950914 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.950856    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e344bb8-5819-4b55-a499-dea024444e95-root\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.950914 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.950878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e344bb8-5819-4b55-a499-dea024444e95-metrics-client-ca\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.950914 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.950911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-tls\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.951181 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.950932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzxc\" (UniqueName: \"kubernetes.io/projected/8e344bb8-5819-4b55-a499-dea024444e95-kube-api-access-5jzxc\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.951181 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.950958    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e344bb8-5819-4b55-a499-dea024444e95-sys\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.951181 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.950976    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-wtmp\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.951181 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.950982    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.951181 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.951073    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e344bb8-5819-4b55-a499-dea024444e95-sys\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.951427 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.951209    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e344bb8-5819-4b55-a499-dea024444e95-root\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.951479 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.951444    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-accelerators-collector-config\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.951542 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.951521    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e344bb8-5819-4b55-a499-dea024444e95-metrics-client-ca\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.951595 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:56:06.951542    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 08 06:56:06.951649 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:56:06.951610    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-tls podName:8e344bb8-5819-4b55-a499-dea024444e95 nodeName:}" failed. No retries permitted until 2026-07-08 06:56:07.45159162 +0000 UTC m=+83.255378002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-tls") pod "node-exporter-4wx47" (UID: "8e344bb8-5819-4b55-a499-dea024444e95") : secret "node-exporter-tls" not found
Jul 08 06:56:06.951903 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.951876    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-textfile\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.953367 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.953345    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:06.975396 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:06.975366    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzxc\" (UniqueName: \"kubernetes.io/projected/8e344bb8-5819-4b55-a499-dea024444e95-kube-api-access-5jzxc\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:07.076885 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:07.076825    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:07.111570 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:07.111535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jvvnp" event={"ID":"3ee5c8ec-83a9-4b40-b2db-25c67997ca4b","Type":"ContainerStarted","Data":"f87fa386741351ce4554d6028993ac5733c428ed712ae26cd86c0b1e78e85ba4"}
Jul 08 06:56:07.186100 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:07.186063    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jvvnp" podStartSLOduration=33.889167258 podStartE2EDuration="43.186051524s" totalImagesPullingTime="9.296884266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:55:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:56:07.182721551 +0000 UTC m=+82.986507951" watchObservedRunningTime="2026-07-08 06:56:07.186051524 +0000 UTC m=+82.989837923"
Jul 08 06:56:07.249319 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:07.249207    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:07.445689 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:07.445661    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:07.454638 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:07.454612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-tls\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:07.457027 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:07.457007    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e344bb8-5819-4b55-a499-dea024444e95-node-exporter-tls\") pod \"node-exporter-4wx47\" (UID: \"8e344bb8-5819-4b55-a499-dea024444e95\") " pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:07.689874 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:07.689850    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4wx47"
Jul 08 06:56:07.704063 ip-10-0-140-110 kubenswrapper[2582]: W0708 06:56:07.704039    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e344bb8_5819_4b55_a499_dea024444e95.slice/crio-d1dc25a0560937ce8d776806fbdeec56142a1561421115e5f0e6fd9b732554b5 WatchSource:0}: Error finding container d1dc25a0560937ce8d776806fbdeec56142a1561421115e5f0e6fd9b732554b5: Status 404 returned error can't find the container with id d1dc25a0560937ce8d776806fbdeec56142a1561421115e5f0e6fd9b732554b5
Jul 08 06:56:07.958816 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:07.958738    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:08.116818 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:08.116788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4wx47" event={"ID":"8e344bb8-5819-4b55-a499-dea024444e95","Type":"ContainerStarted","Data":"d1dc25a0560937ce8d776806fbdeec56142a1561421115e5f0e6fd9b732554b5"}
Jul 08 06:56:08.168288 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:08.168254    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:08.391583 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:08.391510    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:08.691332 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:08.691306    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:09.121414 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:09.121385    2582 generic.go:356] "Generic (PLEG): container finished" podID="8e344bb8-5819-4b55-a499-dea024444e95" containerID="1b9047c7cce176a70243d37c57a70c5a812e5429d471bb6b0ebcf27bc9ce22b6" exitCode=0
Jul 08 06:56:09.121567 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:09.121436    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4wx47" event={"ID":"8e344bb8-5819-4b55-a499-dea024444e95","Type":"ContainerDied","Data":"1b9047c7cce176a70243d37c57a70c5a812e5429d471bb6b0ebcf27bc9ce22b6"}
Jul 08 06:56:09.485501 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:09.485470    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:09.601995 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:09.601967    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:09.705599 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:09.705572    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:10.009125 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:10.009092    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:10.127827 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:10.127788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4wx47" event={"ID":"8e344bb8-5819-4b55-a499-dea024444e95","Type":"ContainerStarted","Data":"2846b2abf7d2742d0a1a8074dd9de2e51a2f70850cccc7d01ac6bd348502352c"}
Jul 08 06:56:10.127974 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:10.127850    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4wx47" event={"ID":"8e344bb8-5819-4b55-a499-dea024444e95","Type":"ContainerStarted","Data":"811e0b0a5f5a583ffaf1e0d6b64bf3b9de6b7e9350846f70aeaf6721aa81af3d"}
Jul 08 06:56:10.231223 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:10.231198    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:10.329095 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:10.329029    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:10.608091 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:10.608028    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:11.386359 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:11.386303    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:11.512966 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:11.512930    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:11.696683 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:11.696646    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:12.640716 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:12.640677    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:12.740417 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:12.740384    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:13.348044 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:13.348008    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:13.474769 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:13.474737    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:14.045875 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:14.045849    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:56:14.066916 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:14.066862    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4wx47" podStartSLOduration=7.081539163 podStartE2EDuration="8.066846602s" totalImagesPullingTime="921.651524ms" totalInitContainerRuntime="63.655915ms" isStatefulPod=true podCreationTimestamp="2026-07-08 06:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:56:10.193817974 +0000 UTC m=+85.997604397" watchObservedRunningTime="2026-07-08 06:56:14.066846602 +0000 UTC m=+89.870633002"
Jul 08 06:56:15.349110 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:15.349076    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:15.450173 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:15.450127    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:17.050865 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:17.050829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:17.152502 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:17.152476    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:17.420440 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:17.420409    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:17.520413 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:17.520384    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:19.064821 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.064780    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c8c549564-96gw6" podUID="40d2fe78-b332-4a62-a021-ab10f43a906f" containerName="registry" containerID="cri-o://de7ab350d4a6a4c7a1217b8cccef37592956ac4762c2490c2627e20a84cf3d06" gracePeriod=30
Jul 08 06:56:19.307307 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.307285    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:56:19.433761 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.433739    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40d2fe78-b332-4a62-a021-ab10f43a906f-ca-trust-extracted\") pod \"40d2fe78-b332-4a62-a021-ab10f43a906f\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") "
Jul 08 06:56:19.433892 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.433783    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwnk8\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-kube-api-access-hwnk8\") pod \"40d2fe78-b332-4a62-a021-ab10f43a906f\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") "
Jul 08 06:56:19.433892 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.433802    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40d2fe78-b332-4a62-a021-ab10f43a906f-trusted-ca\") pod \"40d2fe78-b332-4a62-a021-ab10f43a906f\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") "
Jul 08 06:56:19.433892 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.433841    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-bound-sa-token\") pod \"40d2fe78-b332-4a62-a021-ab10f43a906f\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") "
Jul 08 06:56:19.433892 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.433861    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-certificates\") pod \"40d2fe78-b332-4a62-a021-ab10f43a906f\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") "
Jul 08 06:56:19.433892 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.433880    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40d2fe78-b332-4a62-a021-ab10f43a906f-installation-pull-secrets\") pod \"40d2fe78-b332-4a62-a021-ab10f43a906f\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") "
Jul 08 06:56:19.434130 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.433902    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40d2fe78-b332-4a62-a021-ab10f43a906f-image-registry-private-configuration\") pod \"40d2fe78-b332-4a62-a021-ab10f43a906f\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") "
Jul 08 06:56:19.434130 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.433945    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls\") pod \"40d2fe78-b332-4a62-a021-ab10f43a906f\" (UID: \"40d2fe78-b332-4a62-a021-ab10f43a906f\") "
Jul 08 06:56:19.434522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.434456    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "40d2fe78-b332-4a62-a021-ab10f43a906f" (UID: "40d2fe78-b332-4a62-a021-ab10f43a906f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 06:56:19.434522 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.434475    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d2fe78-b332-4a62-a021-ab10f43a906f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "40d2fe78-b332-4a62-a021-ab10f43a906f" (UID: "40d2fe78-b332-4a62-a021-ab10f43a906f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 06:56:19.436453 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.436426    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "40d2fe78-b332-4a62-a021-ab10f43a906f" (UID: "40d2fe78-b332-4a62-a021-ab10f43a906f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 06:56:19.436571 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.436514    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-kube-api-access-hwnk8" (OuterVolumeSpecName: "kube-api-access-hwnk8") pod "40d2fe78-b332-4a62-a021-ab10f43a906f" (UID: "40d2fe78-b332-4a62-a021-ab10f43a906f"). InnerVolumeSpecName "kube-api-access-hwnk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 06:56:19.436571 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.436557    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "40d2fe78-b332-4a62-a021-ab10f43a906f" (UID: "40d2fe78-b332-4a62-a021-ab10f43a906f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 06:56:19.436687 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.436582    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d2fe78-b332-4a62-a021-ab10f43a906f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "40d2fe78-b332-4a62-a021-ab10f43a906f" (UID: "40d2fe78-b332-4a62-a021-ab10f43a906f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 06:56:19.436687 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.436648    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d2fe78-b332-4a62-a021-ab10f43a906f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "40d2fe78-b332-4a62-a021-ab10f43a906f" (UID: "40d2fe78-b332-4a62-a021-ab10f43a906f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 06:56:19.442373 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.442351    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d2fe78-b332-4a62-a021-ab10f43a906f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "40d2fe78-b332-4a62-a021-ab10f43a906f" (UID: "40d2fe78-b332-4a62-a021-ab10f43a906f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 06:56:19.535165 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.535127    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40d2fe78-b332-4a62-a021-ab10f43a906f-ca-trust-extracted\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 06:56:19.535165 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.535148    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwnk8\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-kube-api-access-hwnk8\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 06:56:19.535279 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.535174    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40d2fe78-b332-4a62-a021-ab10f43a906f-trusted-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 06:56:19.535279 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.535183    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-bound-sa-token\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 06:56:19.535279 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.535191    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-certificates\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 06:56:19.535279 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.535199    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40d2fe78-b332-4a62-a021-ab10f43a906f-installation-pull-secrets\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 06:56:19.535279 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.535209    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40d2fe78-b332-4a62-a021-ab10f43a906f-image-registry-private-configuration\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 06:56:19.535279 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:19.535218    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40d2fe78-b332-4a62-a021-ab10f43a906f-registry-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 06:56:20.162781 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:20.162743    2582 generic.go:356] "Generic (PLEG): container finished" podID="40d2fe78-b332-4a62-a021-ab10f43a906f" containerID="de7ab350d4a6a4c7a1217b8cccef37592956ac4762c2490c2627e20a84cf3d06" exitCode=0
Jul 08 06:56:20.163258 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:20.162788    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8c549564-96gw6"
Jul 08 06:56:20.163258 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:20.162833    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c8c549564-96gw6" event={"ID":"40d2fe78-b332-4a62-a021-ab10f43a906f","Type":"ContainerDied","Data":"de7ab350d4a6a4c7a1217b8cccef37592956ac4762c2490c2627e20a84cf3d06"}
Jul 08 06:56:20.163258 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:20.162894    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c8c549564-96gw6" event={"ID":"40d2fe78-b332-4a62-a021-ab10f43a906f","Type":"ContainerDied","Data":"25f22db1d22e816a7dc67cbe0b67e0276c4d16071b6154aa1b2b0faaf1ab58f4"}
Jul 08 06:56:20.163258 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:20.162927    2582 scope.go:117] "RemoveContainer" containerID="de7ab350d4a6a4c7a1217b8cccef37592956ac4762c2490c2627e20a84cf3d06"
Jul 08 06:56:20.171005 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:20.170991    2582 scope.go:117] "RemoveContainer" containerID="de7ab350d4a6a4c7a1217b8cccef37592956ac4762c2490c2627e20a84cf3d06"
Jul 08 06:56:20.171266 ip-10-0-140-110 kubenswrapper[2582]: E0708 06:56:20.171247    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7ab350d4a6a4c7a1217b8cccef37592956ac4762c2490c2627e20a84cf3d06\": container with ID starting with de7ab350d4a6a4c7a1217b8cccef37592956ac4762c2490c2627e20a84cf3d06 not found: ID does not exist" containerID="de7ab350d4a6a4c7a1217b8cccef37592956ac4762c2490c2627e20a84cf3d06"
Jul 08 06:56:20.171311 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:20.171273    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7ab350d4a6a4c7a1217b8cccef37592956ac4762c2490c2627e20a84cf3d06"} err="failed to get container status \"de7ab350d4a6a4c7a1217b8cccef37592956ac4762c2490c2627e20a84cf3d06\": rpc error: code = NotFound desc = could not find container \"de7ab350d4a6a4c7a1217b8cccef37592956ac4762c2490c2627e20a84cf3d06\": container with ID starting with de7ab350d4a6a4c7a1217b8cccef37592956ac4762c2490c2627e20a84cf3d06 not found: ID does not exist"
Jul 08 06:56:20.183935 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:20.183917    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c8c549564-96gw6"]
Jul 08 06:56:20.187808 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:20.187789    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c8c549564-96gw6"]
Jul 08 06:56:20.366973 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:20.366946    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:20.468712 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:20.468674    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:20.729746 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:20.729680    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:20.744828 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:20.744795    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d2fe78-b332-4a62-a021-ab10f43a906f" path="/var/lib/kubelet/pods/40d2fe78-b332-4a62-a021-ab10f43a906f/volumes"
Jul 08 06:56:20.783042 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:20.783013    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:24.110839 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:24.110806    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:24.211737 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:24.211709    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:24.551860 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:24.551822    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:24.657317 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:24.657283    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:24.754004 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:24.753973    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:24.854210 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:24.854127    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:28.074138 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:28.074109    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qq8lz"
Jul 08 06:56:29.449814 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:29.449777    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:29.504726 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:29.504693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:43.682479 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:43.682430    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:43.758186 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:43.758138    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:47.371080 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:47.371041    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:56:47.422784 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:56:47.422751    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:57:26.245617 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:57:26.245576    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:57:26.447078 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:57:26.447043    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:57:35.655262 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:57:35.655225    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:57:35.756452 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:57:35.756425    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:57:46.878692 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:57:46.878662    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:57:46.978841 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:57:46.978809    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:57:47.078978 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:57:47.078949    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:58:41.495993 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:58:41.495954    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:58:41.597129 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:58:41.597097    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:58:46.625955 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:58:46.625916    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:58:46.725959 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:58:46.725932    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:58:46.888790 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:58:46.888710    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:58:50.388273 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:58:50.388236    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:58:53.468569 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:58:53.468542    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:58:53.571445 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:58:53.571416    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:58:57.152237 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:58:57.152206    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:58:57.256326 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:58:57.256294    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:58:57.322844 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:58:57.322819    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:59:02.381127 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:59:02.381089    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:59:02.480520 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:59:02.480496    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:59:23.935307 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:59:23.935274    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:59:44.609997 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:59:44.609965    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 06:59:44.617860 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:59:44.617835    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 06:59:44.619387 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:59:44.619368    2582 kubelet.go:1648] "Image garbage collection succeeded"
Jul 08 06:59:45.229595 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:59:45.229568    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 06:59:46.465608 ip-10-0-140-110 kubenswrapper[2582]: I0708 06:59:46.465575    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:00:28.450892 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:00:28.450856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:00:28.550652 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:00:28.550619    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:00:34.774128 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:00:34.774093    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:00:35.104089 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:00:35.104012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:00:35.741694 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:00:35.741658    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:00:35.840791 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:00:35.840762    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:00:35.925669 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:00:35.925640    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:00:36.000527 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:00:36.000463    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:00:36.102902 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:00:36.102876    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:00:36.301242 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:00:36.301121    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:00:36.436407 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:00:36.436375    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:00:36.534008 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:00:36.533948    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:00:36.634558 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:00:36.634497    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:00:59.304277 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:00:59.304238    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:01:41.402067 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.402032    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:01:41.462977 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.462949    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-9487895bf-nvzts"]
Jul 08 07:01:41.463205 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.463193    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d2fe78-b332-4a62-a021-ab10f43a906f" containerName="registry"
Jul 08 07:01:41.463266 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.463207    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d2fe78-b332-4a62-a021-ab10f43a906f" containerName="registry"
Jul 08 07:01:41.463307 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.463277    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d2fe78-b332-4a62-a021-ab10f43a906f" containerName="registry"
Jul 08 07:01:41.465856 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.465842    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9487895bf-nvzts"
Jul 08 07:01:41.469578 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.469553    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 08 07:01:41.469578 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.469568    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-w4dd4\""
Jul 08 07:01:41.469774 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.469568    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 08 07:01:41.469774 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.469556    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 08 07:01:41.474813 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.474792    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-9487895bf-nvzts"]
Jul 08 07:01:41.483766 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.483742    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:01:41.511276 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.511252    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-szb56"]
Jul 08 07:01:41.514233 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.514219    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-szb56"
Jul 08 07:01:41.516898 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.516881    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 08 07:01:41.517110 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.517095    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-7pcq6\""
Jul 08 07:01:41.522365 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.522345    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-szb56"]
Jul 08 07:01:41.535199 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.535171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp5q2\" (UniqueName: \"kubernetes.io/projected/812e7060-1958-4e43-8bc7-3c163bd8724e-kube-api-access-mp5q2\") pod \"kserve-controller-manager-9487895bf-nvzts\" (UID: \"812e7060-1958-4e43-8bc7-3c163bd8724e\") " pod="kserve/kserve-controller-manager-9487895bf-nvzts"
Jul 08 07:01:41.535286 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.535218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/812e7060-1958-4e43-8bc7-3c163bd8724e-cert\") pod \"kserve-controller-manager-9487895bf-nvzts\" (UID: \"812e7060-1958-4e43-8bc7-3c163bd8724e\") " pod="kserve/kserve-controller-manager-9487895bf-nvzts"
Jul 08 07:01:41.556005 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.555982    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:01:41.636408 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.636375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp5q2\" (UniqueName: \"kubernetes.io/projected/812e7060-1958-4e43-8bc7-3c163bd8724e-kube-api-access-mp5q2\") pod \"kserve-controller-manager-9487895bf-nvzts\" (UID: \"812e7060-1958-4e43-8bc7-3c163bd8724e\") " pod="kserve/kserve-controller-manager-9487895bf-nvzts"
Jul 08 07:01:41.636526 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.636427    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm9gh\" (UniqueName: \"kubernetes.io/projected/3cb033b1-dc38-4cca-b0a4-95c836e7be0c-kube-api-access-tm9gh\") pod \"seaweedfs-64568bcd49-szb56\" (UID: \"3cb033b1-dc38-4cca-b0a4-95c836e7be0c\") " pod="kserve/seaweedfs-64568bcd49-szb56"
Jul 08 07:01:41.636526 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.636473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/812e7060-1958-4e43-8bc7-3c163bd8724e-cert\") pod \"kserve-controller-manager-9487895bf-nvzts\" (UID: \"812e7060-1958-4e43-8bc7-3c163bd8724e\") " pod="kserve/kserve-controller-manager-9487895bf-nvzts"
Jul 08 07:01:41.636606 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.636522    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3cb033b1-dc38-4cca-b0a4-95c836e7be0c-data\") pod \"seaweedfs-64568bcd49-szb56\" (UID: \"3cb033b1-dc38-4cca-b0a4-95c836e7be0c\") " pod="kserve/seaweedfs-64568bcd49-szb56"
Jul 08 07:01:41.638966 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.638942    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/812e7060-1958-4e43-8bc7-3c163bd8724e-cert\") pod \"kserve-controller-manager-9487895bf-nvzts\" (UID: \"812e7060-1958-4e43-8bc7-3c163bd8724e\") " pod="kserve/kserve-controller-manager-9487895bf-nvzts"
Jul 08 07:01:41.645266 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.645243    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp5q2\" (UniqueName: \"kubernetes.io/projected/812e7060-1958-4e43-8bc7-3c163bd8724e-kube-api-access-mp5q2\") pod \"kserve-controller-manager-9487895bf-nvzts\" (UID: \"812e7060-1958-4e43-8bc7-3c163bd8724e\") " pod="kserve/kserve-controller-manager-9487895bf-nvzts"
Jul 08 07:01:41.737523 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.737490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3cb033b1-dc38-4cca-b0a4-95c836e7be0c-data\") pod \"seaweedfs-64568bcd49-szb56\" (UID: \"3cb033b1-dc38-4cca-b0a4-95c836e7be0c\") " pod="kserve/seaweedfs-64568bcd49-szb56"
Jul 08 07:01:41.737699 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.737550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9gh\" (UniqueName: \"kubernetes.io/projected/3cb033b1-dc38-4cca-b0a4-95c836e7be0c-kube-api-access-tm9gh\") pod \"seaweedfs-64568bcd49-szb56\" (UID: \"3cb033b1-dc38-4cca-b0a4-95c836e7be0c\") " pod="kserve/seaweedfs-64568bcd49-szb56"
Jul 08 07:01:41.737971 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.737948    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3cb033b1-dc38-4cca-b0a4-95c836e7be0c-data\") pod \"seaweedfs-64568bcd49-szb56\" (UID: \"3cb033b1-dc38-4cca-b0a4-95c836e7be0c\") " pod="kserve/seaweedfs-64568bcd49-szb56"
Jul 08 07:01:41.745930 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.745902    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm9gh\" (UniqueName: \"kubernetes.io/projected/3cb033b1-dc38-4cca-b0a4-95c836e7be0c-kube-api-access-tm9gh\") pod \"seaweedfs-64568bcd49-szb56\" (UID: \"3cb033b1-dc38-4cca-b0a4-95c836e7be0c\") " pod="kserve/seaweedfs-64568bcd49-szb56"
Jul 08 07:01:41.776624 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.776589    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9487895bf-nvzts"
Jul 08 07:01:41.823949 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.823746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-szb56"
Jul 08 07:01:41.848834 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.848798    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:01:41.915111 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.915080    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-9487895bf-nvzts"]
Jul 08 07:01:41.917725 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:01:41.917688    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812e7060_1958_4e43_8bc7_3c163bd8724e.slice/crio-76b21fd2b0f85d8908d39c4ba00b4ab6eb994f79d46549437373ee42dac0a0b2 WatchSource:0}: Error finding container 76b21fd2b0f85d8908d39c4ba00b4ab6eb994f79d46549437373ee42dac0a0b2: Status 404 returned error can't find the container with id 76b21fd2b0f85d8908d39c4ba00b4ab6eb994f79d46549437373ee42dac0a0b2
Jul 08 07:01:41.919026 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.919005    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 07:01:41.956348 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:41.956184    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-szb56"]
Jul 08 07:01:41.958775 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:01:41.958748    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb033b1_dc38_4cca_b0a4_95c836e7be0c.slice/crio-9508773ef0551827a6f7310a7965183b59648d3ae26b95346ae64fcf480980cc WatchSource:0}: Error finding container 9508773ef0551827a6f7310a7965183b59648d3ae26b95346ae64fcf480980cc: Status 404 returned error can't find the container with id 9508773ef0551827a6f7310a7965183b59648d3ae26b95346ae64fcf480980cc
Jul 08 07:01:42.182626 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:42.182594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-szb56" event={"ID":"3cb033b1-dc38-4cca-b0a4-95c836e7be0c","Type":"ContainerStarted","Data":"9508773ef0551827a6f7310a7965183b59648d3ae26b95346ae64fcf480980cc"}
Jul 08 07:01:42.183574 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:42.183555    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9487895bf-nvzts" event={"ID":"812e7060-1958-4e43-8bc7-3c163bd8724e","Type":"ContainerStarted","Data":"76b21fd2b0f85d8908d39c4ba00b4ab6eb994f79d46549437373ee42dac0a0b2"}
Jul 08 07:01:42.258320 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:42.258291    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:01:42.359772 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:42.359737    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:01:45.198807 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:45.198764    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9487895bf-nvzts" event={"ID":"812e7060-1958-4e43-8bc7-3c163bd8724e","Type":"ContainerStarted","Data":"c487b866b70ab7dd5c998291714b7d51a62ed01ef93e74917dd14b69eec01f1b"}
Jul 08 07:01:45.199260 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:45.198889    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-9487895bf-nvzts"
Jul 08 07:01:45.951015 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:45.950997    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 08 07:01:46.203598 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:46.203505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-szb56" event={"ID":"3cb033b1-dc38-4cca-b0a4-95c836e7be0c","Type":"ContainerStarted","Data":"a53258738501c1c98ef1c7831d8ced06bf23efefe2e529a961c44ba5b6112d7c"}
Jul 08 07:01:46.204051 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:46.203814    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-szb56"
Jul 08 07:01:46.221423 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:46.221382    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-szb56" podStartSLOduration=1.233140833 podStartE2EDuration="5.221367688s" totalImagesPullingTime="3.988226855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:01:46.220676286 +0000 UTC m=+422.024462687" watchObservedRunningTime="2026-07-08 07:01:46.221367688 +0000 UTC m=+422.025154088"
Jul 08 07:01:46.221909 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:46.221883    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-9487895bf-nvzts" podStartSLOduration=2.280695848 podStartE2EDuration="5.221876916s" totalImagesPullingTime="2.941181068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:01:45.21654242 +0000 UTC m=+421.020328821" watchObservedRunningTime="2026-07-08 07:01:46.221876916 +0000 UTC m=+422.025663324"
Jul 08 07:01:53.213242 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:01:53.213211    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-szb56"
Jul 08 07:02:04.900492 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:04.900455    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:16.209543 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:16.209470    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-9487895bf-nvzts"
Jul 08 07:02:18.996692 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:18.996660    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:19.105467 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:19.105437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:30.144639 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:30.144599    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:30.859411 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:30.859378    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:32.137103 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:32.137075    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jkx7v"]
Jul 08 07:02:32.139067 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:32.139051    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jkx7v"
Jul 08 07:02:32.142083 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:32.142061    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 08 07:02:32.142499 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:32.142481    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-v5kjl\""
Jul 08 07:02:32.142729 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:32.142715    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 08 07:02:32.154382 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:32.154363    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jkx7v"]
Jul 08 07:02:32.285236 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:32.285207    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbba35d2-d780-4bd6-80e7-014fc370a41b-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jkx7v\" (UID: \"cbba35d2-d780-4bd6-80e7-014fc370a41b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jkx7v"
Jul 08 07:02:32.285411 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:32.285249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4mkf\" (UniqueName: \"kubernetes.io/projected/cbba35d2-d780-4bd6-80e7-014fc370a41b-kube-api-access-q4mkf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jkx7v\" (UID: \"cbba35d2-d780-4bd6-80e7-014fc370a41b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jkx7v"
Jul 08 07:02:32.386516 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:32.386488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4mkf\" (UniqueName: \"kubernetes.io/projected/cbba35d2-d780-4bd6-80e7-014fc370a41b-kube-api-access-q4mkf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jkx7v\" (UID: \"cbba35d2-d780-4bd6-80e7-014fc370a41b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jkx7v"
Jul 08 07:02:32.386619 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:32.386539    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbba35d2-d780-4bd6-80e7-014fc370a41b-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jkx7v\" (UID: \"cbba35d2-d780-4bd6-80e7-014fc370a41b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jkx7v"
Jul 08 07:02:32.386890 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:32.386873    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbba35d2-d780-4bd6-80e7-014fc370a41b-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jkx7v\" (UID: \"cbba35d2-d780-4bd6-80e7-014fc370a41b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jkx7v"
Jul 08 07:02:32.397514 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:32.397468    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4mkf\" (UniqueName: \"kubernetes.io/projected/cbba35d2-d780-4bd6-80e7-014fc370a41b-kube-api-access-q4mkf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jkx7v\" (UID: \"cbba35d2-d780-4bd6-80e7-014fc370a41b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jkx7v"
Jul 08 07:02:32.447881 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:32.447861    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jkx7v"
Jul 08 07:02:32.573532 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:32.573510    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jkx7v"]
Jul 08 07:02:32.575351 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:02:32.575323    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbba35d2_d780_4bd6_80e7_014fc370a41b.slice/crio-b2317e30f1f1885ca50d4bbe7cb2f2103d8aa09ce250e4faac67e22411d48ca9 WatchSource:0}: Error finding container b2317e30f1f1885ca50d4bbe7cb2f2103d8aa09ce250e4faac67e22411d48ca9: Status 404 returned error can't find the container with id b2317e30f1f1885ca50d4bbe7cb2f2103d8aa09ce250e4faac67e22411d48ca9
Jul 08 07:02:33.364183 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:33.364129    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jkx7v" event={"ID":"cbba35d2-d780-4bd6-80e7-014fc370a41b","Type":"ContainerStarted","Data":"b2317e30f1f1885ca50d4bbe7cb2f2103d8aa09ce250e4faac67e22411d48ca9"}
Jul 08 07:02:33.959469 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:33.959414    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:34.059349 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:34.059302    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:35.374076 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:35.374035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jkx7v" event={"ID":"cbba35d2-d780-4bd6-80e7-014fc370a41b","Type":"ContainerStarted","Data":"520efe36afe8b763e10746ba32a471fd14598d7f49a99f6518bf1edd455a2b49"}
Jul 08 07:02:35.461732 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:35.461694    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:35.561010 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:35.560972    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:37.920419 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:37.920384    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:38.246428 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:38.246346    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:38.745717 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:38.745682    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:38.803922 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:38.803894    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:40.452548 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:40.452512    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:40.602242 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:40.602210    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:40.698218 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:40.698110    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:41.402820 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:41.402786    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:41.472048 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:41.472019    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:41.582722 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:41.582692    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:41.638984 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:41.638964    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:42.130782 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:42.130748    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:43.805169 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:43.805099    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jkx7v" podStartSLOduration=9.621224664 podStartE2EDuration="11.805082118s" totalImagesPullingTime="2.183857454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:02:35.405515022 +0000 UTC m=+471.209301423" watchObservedRunningTime="2026-07-08 07:02:43.805082118 +0000 UTC m=+479.608868517"
Jul 08 07:02:43.805799 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:43.805778    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5lmzt"]
Jul 08 07:02:43.808884 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:43.808868    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5lmzt"
Jul 08 07:02:43.811514 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:43.811486    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 08 07:02:43.811643 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:43.811565    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 08 07:02:43.811643 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:43.811576    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jzwkh\""
Jul 08 07:02:43.818181 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:43.818142    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5lmzt"]
Jul 08 07:02:43.862798 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:43.862777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5wv5\" (UniqueName: \"kubernetes.io/projected/578f0713-57f2-4d21-9d82-c1de642bb616-kube-api-access-b5wv5\") pod \"cert-manager-cainjector-559cdd8c87-5lmzt\" (UID: \"578f0713-57f2-4d21-9d82-c1de642bb616\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5lmzt"
Jul 08 07:02:43.862889 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:43.862815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/578f0713-57f2-4d21-9d82-c1de642bb616-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5lmzt\" (UID: \"578f0713-57f2-4d21-9d82-c1de642bb616\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5lmzt"
Jul 08 07:02:43.963321 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:43.963293    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5wv5\" (UniqueName: \"kubernetes.io/projected/578f0713-57f2-4d21-9d82-c1de642bb616-kube-api-access-b5wv5\") pod \"cert-manager-cainjector-559cdd8c87-5lmzt\" (UID: \"578f0713-57f2-4d21-9d82-c1de642bb616\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5lmzt"
Jul 08 07:02:43.963407 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:43.963335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/578f0713-57f2-4d21-9d82-c1de642bb616-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5lmzt\" (UID: \"578f0713-57f2-4d21-9d82-c1de642bb616\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5lmzt"
Jul 08 07:02:43.973034 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:43.973014    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/578f0713-57f2-4d21-9d82-c1de642bb616-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5lmzt\" (UID: \"578f0713-57f2-4d21-9d82-c1de642bb616\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5lmzt"
Jul 08 07:02:43.973129 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:43.973107    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5wv5\" (UniqueName: \"kubernetes.io/projected/578f0713-57f2-4d21-9d82-c1de642bb616-kube-api-access-b5wv5\") pod \"cert-manager-cainjector-559cdd8c87-5lmzt\" (UID: \"578f0713-57f2-4d21-9d82-c1de642bb616\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5lmzt"
Jul 08 07:02:44.118228 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:44.118136    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5lmzt"
Jul 08 07:02:44.239721 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:44.239692    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5lmzt"]
Jul 08 07:02:44.242641 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:02:44.242609    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod578f0713_57f2_4d21_9d82_c1de642bb616.slice/crio-3c4fc44d4ab96421b63dd910c133b48128787a7afaf17657ae77b10e5e83e22d WatchSource:0}: Error finding container 3c4fc44d4ab96421b63dd910c133b48128787a7afaf17657ae77b10e5e83e22d: Status 404 returned error can't find the container with id 3c4fc44d4ab96421b63dd910c133b48128787a7afaf17657ae77b10e5e83e22d
Jul 08 07:02:44.405400 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:44.405336    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5lmzt" event={"ID":"578f0713-57f2-4d21-9d82-c1de642bb616","Type":"ContainerStarted","Data":"3c4fc44d4ab96421b63dd910c133b48128787a7afaf17657ae77b10e5e83e22d"}
Jul 08 07:02:45.455146 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:45.455112    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:46.910771 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:46.910737    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ldzjl"]
Jul 08 07:02:46.914141 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:46.914121    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ldzjl"
Jul 08 07:02:46.916575 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:46.916556    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xlrf6\""
Jul 08 07:02:46.922698 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:46.922673    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ldzjl"]
Jul 08 07:02:46.983421 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:46.983391    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d1a7697-04ee-42af-bb95-916fa1f80734-bound-sa-token\") pod \"cert-manager-55b4d954d5-ldzjl\" (UID: \"0d1a7697-04ee-42af-bb95-916fa1f80734\") " pod="cert-manager/cert-manager-55b4d954d5-ldzjl"
Jul 08 07:02:46.983554 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:46.983441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm2v4\" (UniqueName: \"kubernetes.io/projected/0d1a7697-04ee-42af-bb95-916fa1f80734-kube-api-access-tm2v4\") pod \"cert-manager-55b4d954d5-ldzjl\" (UID: \"0d1a7697-04ee-42af-bb95-916fa1f80734\") " pod="cert-manager/cert-manager-55b4d954d5-ldzjl"
Jul 08 07:02:47.084761 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:47.084686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d1a7697-04ee-42af-bb95-916fa1f80734-bound-sa-token\") pod \"cert-manager-55b4d954d5-ldzjl\" (UID: \"0d1a7697-04ee-42af-bb95-916fa1f80734\") " pod="cert-manager/cert-manager-55b4d954d5-ldzjl"
Jul 08 07:02:47.084761 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:47.084756    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm2v4\" (UniqueName: \"kubernetes.io/projected/0d1a7697-04ee-42af-bb95-916fa1f80734-kube-api-access-tm2v4\") pod \"cert-manager-55b4d954d5-ldzjl\" (UID: \"0d1a7697-04ee-42af-bb95-916fa1f80734\") " pod="cert-manager/cert-manager-55b4d954d5-ldzjl"
Jul 08 07:02:47.102432 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:47.102401    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d1a7697-04ee-42af-bb95-916fa1f80734-bound-sa-token\") pod \"cert-manager-55b4d954d5-ldzjl\" (UID: \"0d1a7697-04ee-42af-bb95-916fa1f80734\") " pod="cert-manager/cert-manager-55b4d954d5-ldzjl"
Jul 08 07:02:47.104188 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:47.104168    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm2v4\" (UniqueName: \"kubernetes.io/projected/0d1a7697-04ee-42af-bb95-916fa1f80734-kube-api-access-tm2v4\") pod \"cert-manager-55b4d954d5-ldzjl\" (UID: \"0d1a7697-04ee-42af-bb95-916fa1f80734\") " pod="cert-manager/cert-manager-55b4d954d5-ldzjl"
Jul 08 07:02:47.223092 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:47.223061    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ldzjl"
Jul 08 07:02:47.359115 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:47.359023    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ldzjl"]
Jul 08 07:02:47.363775 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:02:47.363750    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d1a7697_04ee_42af_bb95_916fa1f80734.slice/crio-59e359e4657cf539074d509faa288103f4b5608032f98a611181dd2b1946977d WatchSource:0}: Error finding container 59e359e4657cf539074d509faa288103f4b5608032f98a611181dd2b1946977d: Status 404 returned error can't find the container with id 59e359e4657cf539074d509faa288103f4b5608032f98a611181dd2b1946977d
Jul 08 07:02:47.425620 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:47.425567    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5lmzt" event={"ID":"578f0713-57f2-4d21-9d82-c1de642bb616","Type":"ContainerStarted","Data":"c48a6ffb48793e4e842b2570b1736bb120b0f90a8db384131bff503539b209c8"}
Jul 08 07:02:47.427289 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:47.427263    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ldzjl" event={"ID":"0d1a7697-04ee-42af-bb95-916fa1f80734","Type":"ContainerStarted","Data":"59e359e4657cf539074d509faa288103f4b5608032f98a611181dd2b1946977d"}
Jul 08 07:02:47.441565 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:47.441527    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5lmzt" podStartSLOduration=1.8968839979999998 podStartE2EDuration="4.441512937s" totalImagesPullingTime="2.544628939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:02:47.440390799 +0000 UTC m=+483.244177195" watchObservedRunningTime="2026-07-08 07:02:47.441512937 +0000 UTC m=+483.245299348"
Jul 08 07:02:47.757448 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:47.757410    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:47.858986 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:47.858952    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:48.433047 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:48.433008    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ldzjl" event={"ID":"0d1a7697-04ee-42af-bb95-916fa1f80734","Type":"ContainerStarted","Data":"92a7e172d8b93770677de59a59e01bdaf7d14dc08a9953ce0bf86af4c501c849"}
Jul 08 07:02:48.449435 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:48.449392    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ldzjl" podStartSLOduration=2.449379044 podStartE2EDuration="2.449379044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:02:48.448140502 +0000 UTC m=+484.251926913" watchObservedRunningTime="2026-07-08 07:02:48.449379044 +0000 UTC m=+484.253165443"
Jul 08 07:02:52.223112 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:52.223073    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:52.318454 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:52.318423    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:52.375921 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:52.375885    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:52.554033 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:52.553967    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:52.654109 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:52.654087    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:53.739941 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:53.739908    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:02:53.790743 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:02:53.790713    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:03.840848 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:03.840822    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:08.822413 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:08.822373    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:08.922286 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:08.922259    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:09.725443 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:09.725412    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:12.312861 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:12.312826    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:12.372674 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:12.372640    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:15.198235 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.198200    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"]
Jul 08 07:03:15.203004 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.202976    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.206213 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.206185    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 08 07:03:15.206340 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.206226    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 08 07:03:15.206340 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.206318    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-67g49\""
Jul 08 07:03:15.206589 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.206570    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 08 07:03:15.215518 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.215232    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"]
Jul 08 07:03:15.243849 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.243811    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:15.283016 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.282988    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-workload-socket\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.283172 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.283020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istio-token\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.283172 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.283045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-workload-certs\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.283172 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.283093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-credential-socket\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.283172 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.283170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istio-data\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.283332 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.283211    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istio-envoy\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.283332 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.283225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.283332 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.283244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzkgv\" (UniqueName: \"kubernetes.io/projected/c1e1a16a-a06e-4b84-842c-6b4415e0f636-kube-api-access-bzkgv\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.283332 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.283261    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.325685 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.325659    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:15.383876 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.383850    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istio-envoy\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.383876 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.383876    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.384039 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.383893    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzkgv\" (UniqueName: \"kubernetes.io/projected/c1e1a16a-a06e-4b84-842c-6b4415e0f636-kube-api-access-bzkgv\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.384039 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.383910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.384039 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.383927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-workload-socket\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.384039 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.383947    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istio-token\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.384039 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.383978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-workload-certs\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.384039 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.384016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-credential-socket\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.384352 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.384051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istio-data\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.384443 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.384417    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-workload-socket\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.384502 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.384477    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-credential-socket\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.384601 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.384575    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istio-data\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.384777 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.384758    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.384839 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.384816    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-workload-certs\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.386297 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.386279    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istio-envoy\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.386417 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.386402    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.391692 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.391645    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e1a16a-a06e-4b84-842c-6b4415e0f636-istio-token\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.391863 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.391842    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzkgv\" (UniqueName: \"kubernetes.io/projected/c1e1a16a-a06e-4b84-842c-6b4415e0f636-kube-api-access-bzkgv\") pod \"openshift-ai-inference-openshift-default-9455cf8d5-g22cx\" (UID: \"c1e1a16a-a06e-4b84-842c-6b4415e0f636\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.514516 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.514460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:15.648131 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:15.648092    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"]
Jul 08 07:03:15.650928 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:03:15.650898    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e1a16a_a06e_4b84_842c_6b4415e0f636.slice/crio-74dd427891660b699f4e2bc0f424f704131237daa6ddf0875edaed465f9d52c1 WatchSource:0}: Error finding container 74dd427891660b699f4e2bc0f424f704131237daa6ddf0875edaed465f9d52c1: Status 404 returned error can't find the container with id 74dd427891660b699f4e2bc0f424f704131237daa6ddf0875edaed465f9d52c1
Jul 08 07:03:16.549747 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:16.549704    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerStarted","Data":"74dd427891660b699f4e2bc0f424f704131237daa6ddf0875edaed465f9d52c1"}
Jul 08 07:03:18.254039 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:18.253997    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:03:18.254352 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:18.254075    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:03:18.254352 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:18.254105    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:03:18.523113 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:18.523083    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:18.558766 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:18.558732    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerStarted","Data":"36c3a6e0cf7013208444f6739eb265949fe41b00dd4324d500c56c2c225d2206"}
Jul 08 07:03:18.579706 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:18.579665    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podStartSLOduration=0.978849084 podStartE2EDuration="3.579642863s" totalImagesPullingTime="2.600793779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:03:18.578431966 +0000 UTC m=+514.382218367" watchObservedRunningTime="2026-07-08 07:03:18.579642863 +0000 UTC m=+514.383429263"
Jul 08 07:03:19.515225 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:19.515192    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:19.519970 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:19.519944    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:19.562791 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:19.562771    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:19.563563 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:19.563545    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:03:20.419664 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.419625    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:20.431984 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.431960    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"]
Jul 08 07:03:20.434655 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.434640    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:20.438694 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.438671    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 08 07:03:20.438694 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.438689    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 08 07:03:20.438888 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.438717    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 08 07:03:20.438888 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.438671    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nh4tx\""
Jul 08 07:03:20.438888 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.438672    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 08 07:03:20.439048 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.438970    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 08 07:03:20.446100 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.446076    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"]
Jul 08 07:03:20.507391 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.507365    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:20.524513 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.524491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a075109-5098-4ab8-8b73-2e85e402401a-metrics-cert\") pod \"lws-controller-manager-58c59b5fc8-lbghs\" (UID: \"5a075109-5098-4ab8-8b73-2e85e402401a\") " pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:20.524822 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.524521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a075109-5098-4ab8-8b73-2e85e402401a-cert\") pod \"lws-controller-manager-58c59b5fc8-lbghs\" (UID: \"5a075109-5098-4ab8-8b73-2e85e402401a\") " pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:20.524822 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.524543    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5a075109-5098-4ab8-8b73-2e85e402401a-manager-config\") pod \"lws-controller-manager-58c59b5fc8-lbghs\" (UID: \"5a075109-5098-4ab8-8b73-2e85e402401a\") " pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:20.524822 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.524575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z8fl\" (UniqueName: \"kubernetes.io/projected/5a075109-5098-4ab8-8b73-2e85e402401a-kube-api-access-9z8fl\") pod \"lws-controller-manager-58c59b5fc8-lbghs\" (UID: \"5a075109-5098-4ab8-8b73-2e85e402401a\") " pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:20.625469 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.625445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a075109-5098-4ab8-8b73-2e85e402401a-metrics-cert\") pod \"lws-controller-manager-58c59b5fc8-lbghs\" (UID: \"5a075109-5098-4ab8-8b73-2e85e402401a\") " pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:20.625567 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.625473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a075109-5098-4ab8-8b73-2e85e402401a-cert\") pod \"lws-controller-manager-58c59b5fc8-lbghs\" (UID: \"5a075109-5098-4ab8-8b73-2e85e402401a\") " pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:20.625567 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.625494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5a075109-5098-4ab8-8b73-2e85e402401a-manager-config\") pod \"lws-controller-manager-58c59b5fc8-lbghs\" (UID: \"5a075109-5098-4ab8-8b73-2e85e402401a\") " pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:20.625635 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.625598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z8fl\" (UniqueName: \"kubernetes.io/projected/5a075109-5098-4ab8-8b73-2e85e402401a-kube-api-access-9z8fl\") pod \"lws-controller-manager-58c59b5fc8-lbghs\" (UID: \"5a075109-5098-4ab8-8b73-2e85e402401a\") " pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:20.626198 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.626176    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5a075109-5098-4ab8-8b73-2e85e402401a-manager-config\") pod \"lws-controller-manager-58c59b5fc8-lbghs\" (UID: \"5a075109-5098-4ab8-8b73-2e85e402401a\") " pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:20.627988 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.627968    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a075109-5098-4ab8-8b73-2e85e402401a-cert\") pod \"lws-controller-manager-58c59b5fc8-lbghs\" (UID: \"5a075109-5098-4ab8-8b73-2e85e402401a\") " pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:20.628067 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.628032    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a075109-5098-4ab8-8b73-2e85e402401a-metrics-cert\") pod \"lws-controller-manager-58c59b5fc8-lbghs\" (UID: \"5a075109-5098-4ab8-8b73-2e85e402401a\") " pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:20.635912 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.635888    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z8fl\" (UniqueName: \"kubernetes.io/projected/5a075109-5098-4ab8-8b73-2e85e402401a-kube-api-access-9z8fl\") pod \"lws-controller-manager-58c59b5fc8-lbghs\" (UID: \"5a075109-5098-4ab8-8b73-2e85e402401a\") " pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:20.744360 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.744147    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:20.751089 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.751063    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:20.856055 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.855973    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:20.879992 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:20.879970    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"]
Jul 08 07:03:20.882430 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:03:20.882404    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a075109_5098_4ab8_8b73_2e85e402401a.slice/crio-35831861bfe016de0c33b7235b6c23a5cd7b93414fb0c248070761f7d86787a4 WatchSource:0}: Error finding container 35831861bfe016de0c33b7235b6c23a5cd7b93414fb0c248070761f7d86787a4: Status 404 returned error can't find the container with id 35831861bfe016de0c33b7235b6c23a5cd7b93414fb0c248070761f7d86787a4
Jul 08 07:03:21.571752 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:21.571707    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs" event={"ID":"5a075109-5098-4ab8-8b73-2e85e402401a","Type":"ContainerStarted","Data":"35831861bfe016de0c33b7235b6c23a5cd7b93414fb0c248070761f7d86787a4"}
Jul 08 07:03:24.587430 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:24.587387    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs" event={"ID":"5a075109-5098-4ab8-8b73-2e85e402401a","Type":"ContainerStarted","Data":"78a47c710814593fb7d04cb216e457b2f96388639616db7802c69dd1f849e1d1"}
Jul 08 07:03:24.587865 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:24.587529    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:24.604782 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:24.604740    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs" podStartSLOduration=1.364073037 podStartE2EDuration="4.604729642s" totalImagesPullingTime="3.240656605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:03:24.603922705 +0000 UTC m=+520.407709128" watchObservedRunningTime="2026-07-08 07:03:24.604729642 +0000 UTC m=+520.408516041"
Jul 08 07:03:35.593075 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:35.593047    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58c59b5fc8-lbghs"
Jul 08 07:03:56.135812 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:56.135779    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:03:57.135855 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:03:57.135826    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:01.558055 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:01.558023    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:02.360226 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:02.360191    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:02.636612 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:02.636531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:02.800631 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:02.800599    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:02.955960 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:02.955932    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:06.419554 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:06.419521    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:06.518311 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:06.518285    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:07.965718 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:07.965687    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:09.004145 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:09.004112    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:10.152768 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:10.152738    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:10.406817 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:10.406734    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:13.213016 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:13.212988    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:13.356498 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:13.356467    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:13.456921 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:13.456890    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:18.742051 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:18.742015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:18.802724 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:18.802693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:18.943952 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:18.943927    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:19.006969 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:19.006917    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:19.158023 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:19.158001    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:19.223495 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:19.223472    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:20.273729 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.273695    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"]
Jul 08 07:04:20.274396 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.274369    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:20.277577 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.277124    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"
Jul 08 07:04:20.279882 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.279859    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 08 07:04:20.280000 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.279910    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 08 07:04:20.280000 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.279993    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mps7s\""
Jul 08 07:04:20.280971 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.280956    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 08 07:04:20.281072 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.280989    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 08 07:04:20.284812 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.284790    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"]
Jul 08 07:04:20.343106 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.343081    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c4102a-0424-46a9-822c-2057ea98f8c9-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-8r75k\" (UID: \"b5c4102a-0424-46a9-822c-2057ea98f8c9\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"
Jul 08 07:04:20.343255 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.343123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5c4102a-0424-46a9-822c-2057ea98f8c9-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-8r75k\" (UID: \"b5c4102a-0424-46a9-822c-2057ea98f8c9\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"
Jul 08 07:04:20.343255 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.343140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2kwh\" (UniqueName: \"kubernetes.io/projected/b5c4102a-0424-46a9-822c-2057ea98f8c9-kube-api-access-h2kwh\") pod \"kuadrant-console-plugin-85fd4b8b9d-8r75k\" (UID: \"b5c4102a-0424-46a9-822c-2057ea98f8c9\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"
Jul 08 07:04:20.443832 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.443807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c4102a-0424-46a9-822c-2057ea98f8c9-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-8r75k\" (UID: \"b5c4102a-0424-46a9-822c-2057ea98f8c9\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"
Jul 08 07:04:20.443941 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.443870    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5c4102a-0424-46a9-822c-2057ea98f8c9-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-8r75k\" (UID: \"b5c4102a-0424-46a9-822c-2057ea98f8c9\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"
Jul 08 07:04:20.443941 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.443902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2kwh\" (UniqueName: \"kubernetes.io/projected/b5c4102a-0424-46a9-822c-2057ea98f8c9-kube-api-access-h2kwh\") pod \"kuadrant-console-plugin-85fd4b8b9d-8r75k\" (UID: \"b5c4102a-0424-46a9-822c-2057ea98f8c9\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"
Jul 08 07:04:20.444025 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:04:20.443972    2582 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 08 07:04:20.444061 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:04:20.444048    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c4102a-0424-46a9-822c-2057ea98f8c9-plugin-serving-cert podName:b5c4102a-0424-46a9-822c-2057ea98f8c9 nodeName:}" failed. No retries permitted until 2026-07-08 07:04:20.944027929 +0000 UTC m=+576.747814321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b5c4102a-0424-46a9-822c-2057ea98f8c9-plugin-serving-cert") pod "kuadrant-console-plugin-85fd4b8b9d-8r75k" (UID: "b5c4102a-0424-46a9-822c-2057ea98f8c9") : secret "plugin-serving-cert" not found
Jul 08 07:04:20.444564 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.444546    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5c4102a-0424-46a9-822c-2057ea98f8c9-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-8r75k\" (UID: \"b5c4102a-0424-46a9-822c-2057ea98f8c9\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"
Jul 08 07:04:20.452482 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.452460    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2kwh\" (UniqueName: \"kubernetes.io/projected/b5c4102a-0424-46a9-822c-2057ea98f8c9-kube-api-access-h2kwh\") pod \"kuadrant-console-plugin-85fd4b8b9d-8r75k\" (UID: \"b5c4102a-0424-46a9-822c-2057ea98f8c9\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"
Jul 08 07:04:20.946871 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:20.946842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c4102a-0424-46a9-822c-2057ea98f8c9-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-8r75k\" (UID: \"b5c4102a-0424-46a9-822c-2057ea98f8c9\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"
Jul 08 07:04:20.947030 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:04:20.946953    2582 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 08 07:04:20.947030 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:04:20.947009    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c4102a-0424-46a9-822c-2057ea98f8c9-plugin-serving-cert podName:b5c4102a-0424-46a9-822c-2057ea98f8c9 nodeName:}" failed. No retries permitted until 2026-07-08 07:04:21.946995085 +0000 UTC m=+577.750781463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b5c4102a-0424-46a9-822c-2057ea98f8c9-plugin-serving-cert") pod "kuadrant-console-plugin-85fd4b8b9d-8r75k" (UID: "b5c4102a-0424-46a9-822c-2057ea98f8c9") : secret "plugin-serving-cert" not found
Jul 08 07:04:21.423892 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:21.423862    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:21.953832 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:21.953803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c4102a-0424-46a9-822c-2057ea98f8c9-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-8r75k\" (UID: \"b5c4102a-0424-46a9-822c-2057ea98f8c9\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"
Jul 08 07:04:21.956222 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:21.956186    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c4102a-0424-46a9-822c-2057ea98f8c9-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-8r75k\" (UID: \"b5c4102a-0424-46a9-822c-2057ea98f8c9\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"
Jul 08 07:04:22.088688 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:22.088662    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"
Jul 08 07:04:22.215217 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:22.215047    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k"]
Jul 08 07:04:22.217773 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:04:22.217739    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5c4102a_0424_46a9_822c_2057ea98f8c9.slice/crio-ca99d90b3bcdf32973799f1111070d463eb06a3d4f3c87f56388c08a460859a8 WatchSource:0}: Error finding container ca99d90b3bcdf32973799f1111070d463eb06a3d4f3c87f56388c08a460859a8: Status 404 returned error can't find the container with id ca99d90b3bcdf32973799f1111070d463eb06a3d4f3c87f56388c08a460859a8
Jul 08 07:04:22.809448 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:22.809414    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k" event={"ID":"b5c4102a-0424-46a9-822c-2057ea98f8c9","Type":"ContainerStarted","Data":"ca99d90b3bcdf32973799f1111070d463eb06a3d4f3c87f56388c08a460859a8"}
Jul 08 07:04:25.050322 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:25.050281    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:25.149454 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:25.149421    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:26.828633 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:26.828597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k" event={"ID":"b5c4102a-0424-46a9-822c-2057ea98f8c9","Type":"ContainerStarted","Data":"46630e781874b1282336f8acc47f62d56646f4538cbeb6355616069a829bba69"}
Jul 08 07:04:26.847629 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:26.847590    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-8r75k" podStartSLOduration=2.372364767 podStartE2EDuration="6.847576257s" totalImagesPullingTime="4.47521149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:04:26.846404459 +0000 UTC m=+582.650190879" watchObservedRunningTime="2026-07-08 07:04:26.847576257 +0000 UTC m=+582.651362656"
Jul 08 07:04:29.891125 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:29.891092    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:30.054137 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:30.054108    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:33.592946 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:33.592913    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:33.693504 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:33.693475    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:34.193716 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:34.193679    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:34.293723 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:34.293693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:39.593767 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:39.593731    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:39.693257 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:39.693231    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:40.188652 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:40.188618    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:40.251884 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:40.251858    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:40.803707 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:40.803672    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:40.860935 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:40.860904    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:41.082949 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:41.082865    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:44.634811 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:44.634783    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:04:44.640174 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:44.640129    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:04:46.526443 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:46.526411    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:49.821673 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:49.821646    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:51.801657 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:51.801620    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:51.901593 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:51.901563    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:04:56.366667 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:04:56.366636    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:05:04.250756 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:04.250723    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:05:04.360803 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:04.360767    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:05:04.461021 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:04.460988    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:05:05.057280 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:05.057248    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c89bb85-kntfg"]
Jul 08 07:05:05.083116 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:05.083067    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-kntfg"]
Jul 08 07:05:05.083282 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:05.083229    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-kntfg"
Jul 08 07:05:05.085869 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:05.085851    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cxt5w\""
Jul 08 07:05:05.091420 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:05.091399    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:05:05.161622 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:05.161599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cn2f\" (UniqueName: \"kubernetes.io/projected/3dfc67dd-d6b3-494e-b690-c2c17818c048-kube-api-access-5cn2f\") pod \"authorino-b6c89bb85-kntfg\" (UID: \"3dfc67dd-d6b3-494e-b690-c2c17818c048\") " pod="kuadrant-system/authorino-b6c89bb85-kntfg"
Jul 08 07:05:05.191105 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:05.191084    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:05:05.262327 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:05.262296    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cn2f\" (UniqueName: \"kubernetes.io/projected/3dfc67dd-d6b3-494e-b690-c2c17818c048-kube-api-access-5cn2f\") pod \"authorino-b6c89bb85-kntfg\" (UID: \"3dfc67dd-d6b3-494e-b690-c2c17818c048\") " pod="kuadrant-system/authorino-b6c89bb85-kntfg"
Jul 08 07:05:05.270005 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:05.269983    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cn2f\" (UniqueName: \"kubernetes.io/projected/3dfc67dd-d6b3-494e-b690-c2c17818c048-kube-api-access-5cn2f\") pod \"authorino-b6c89bb85-kntfg\" (UID: \"3dfc67dd-d6b3-494e-b690-c2c17818c048\") " pod="kuadrant-system/authorino-b6c89bb85-kntfg"
Jul 08 07:05:05.391556 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:05.391500    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-kntfg"
Jul 08 07:05:05.514555 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:05.514519    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-kntfg"]
Jul 08 07:05:05.517433 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:05:05.517402    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dfc67dd_d6b3_494e_b690_c2c17818c048.slice/crio-12b3512ab2313f15e00214c7e9a5d38075677d60313766d44c590b0dd3f2e3d7 WatchSource:0}: Error finding container 12b3512ab2313f15e00214c7e9a5d38075677d60313766d44c590b0dd3f2e3d7: Status 404 returned error can't find the container with id 12b3512ab2313f15e00214c7e9a5d38075677d60313766d44c590b0dd3f2e3d7
Jul 08 07:05:05.979012 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:05.978983    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-kntfg" event={"ID":"3dfc67dd-d6b3-494e-b690-c2c17818c048","Type":"ContainerStarted","Data":"12b3512ab2313f15e00214c7e9a5d38075677d60313766d44c590b0dd3f2e3d7"}
Jul 08 07:05:08.991680 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:08.991648    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-kntfg" event={"ID":"3dfc67dd-d6b3-494e-b690-c2c17818c048","Type":"ContainerStarted","Data":"276c340598bc7ea74f1a0bb56846b739cb2723cbe9cd3a96cbdbf2ff9c33b84a"}
Jul 08 07:05:09.013336 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:09.013281    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c89bb85-kntfg" podStartSLOduration=1.07739345 podStartE2EDuration="4.013266898s" totalImagesPullingTime="2.935873448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:05:09.013094538 +0000 UTC m=+624.816880951" watchObservedRunningTime="2026-07-08 07:05:09.013266898 +0000 UTC m=+624.817053299"
Jul 08 07:05:10.747595 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:10.747565    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-kntfg"]
Jul 08 07:05:10.999344 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:10.999258    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c89bb85-kntfg" podUID="3dfc67dd-d6b3-494e-b690-c2c17818c048" containerName="authorino" containerID="cri-o://276c340598bc7ea74f1a0bb56846b739cb2723cbe9cd3a96cbdbf2ff9c33b84a" gracePeriod=30
Jul 08 07:05:11.239289 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:11.239264    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-kntfg"
Jul 08 07:05:11.308215 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:11.308128    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cn2f\" (UniqueName: \"kubernetes.io/projected/3dfc67dd-d6b3-494e-b690-c2c17818c048-kube-api-access-5cn2f\") pod \"3dfc67dd-d6b3-494e-b690-c2c17818c048\" (UID: \"3dfc67dd-d6b3-494e-b690-c2c17818c048\") "
Jul 08 07:05:11.310128 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:11.310100    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfc67dd-d6b3-494e-b690-c2c17818c048-kube-api-access-5cn2f" (OuterVolumeSpecName: "kube-api-access-5cn2f") pod "3dfc67dd-d6b3-494e-b690-c2c17818c048" (UID: "3dfc67dd-d6b3-494e-b690-c2c17818c048"). InnerVolumeSpecName "kube-api-access-5cn2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:05:11.409275 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:11.409254    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cn2f\" (UniqueName: \"kubernetes.io/projected/3dfc67dd-d6b3-494e-b690-c2c17818c048-kube-api-access-5cn2f\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:05:12.004334 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:12.004304    2582 generic.go:356] "Generic (PLEG): container finished" podID="3dfc67dd-d6b3-494e-b690-c2c17818c048" containerID="276c340598bc7ea74f1a0bb56846b739cb2723cbe9cd3a96cbdbf2ff9c33b84a" exitCode=0
Jul 08 07:05:12.004723 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:12.004341    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-kntfg"
Jul 08 07:05:12.004723 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:12.004371    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-kntfg" event={"ID":"3dfc67dd-d6b3-494e-b690-c2c17818c048","Type":"ContainerDied","Data":"276c340598bc7ea74f1a0bb56846b739cb2723cbe9cd3a96cbdbf2ff9c33b84a"}
Jul 08 07:05:12.004723 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:12.004408    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-kntfg" event={"ID":"3dfc67dd-d6b3-494e-b690-c2c17818c048","Type":"ContainerDied","Data":"12b3512ab2313f15e00214c7e9a5d38075677d60313766d44c590b0dd3f2e3d7"}
Jul 08 07:05:12.004723 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:12.004425    2582 scope.go:117] "RemoveContainer" containerID="276c340598bc7ea74f1a0bb56846b739cb2723cbe9cd3a96cbdbf2ff9c33b84a"
Jul 08 07:05:12.012886 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:12.012870    2582 scope.go:117] "RemoveContainer" containerID="276c340598bc7ea74f1a0bb56846b739cb2723cbe9cd3a96cbdbf2ff9c33b84a"
Jul 08 07:05:12.013121 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:05:12.013103    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"276c340598bc7ea74f1a0bb56846b739cb2723cbe9cd3a96cbdbf2ff9c33b84a\": container with ID starting with 276c340598bc7ea74f1a0bb56846b739cb2723cbe9cd3a96cbdbf2ff9c33b84a not found: ID does not exist" containerID="276c340598bc7ea74f1a0bb56846b739cb2723cbe9cd3a96cbdbf2ff9c33b84a"
Jul 08 07:05:12.013192 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:12.013129    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"276c340598bc7ea74f1a0bb56846b739cb2723cbe9cd3a96cbdbf2ff9c33b84a"} err="failed to get container status \"276c340598bc7ea74f1a0bb56846b739cb2723cbe9cd3a96cbdbf2ff9c33b84a\": rpc error: code = NotFound desc = could not find container \"276c340598bc7ea74f1a0bb56846b739cb2723cbe9cd3a96cbdbf2ff9c33b84a\": container with ID starting with 276c340598bc7ea74f1a0bb56846b739cb2723cbe9cd3a96cbdbf2ff9c33b84a not found: ID does not exist"
Jul 08 07:05:12.026110 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:12.026089    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-kntfg"]
Jul 08 07:05:12.031778 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:12.031757    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-kntfg"]
Jul 08 07:05:12.745417 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:12.745379    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dfc67dd-d6b3-494e-b690-c2c17818c048" path="/var/lib/kubelet/pods/3dfc67dd-d6b3-494e-b690-c2c17818c048/volumes"
Jul 08 07:05:26.832613 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:26.832577    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:05:27.118449 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:27.118371    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:05:30.591168 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.591110    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-5bxlp"]
Jul 08 07:05:30.591538 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.591428    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dfc67dd-d6b3-494e-b690-c2c17818c048" containerName="authorino"
Jul 08 07:05:30.591538 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.591444    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dfc67dd-d6b3-494e-b690-c2c17818c048" containerName="authorino"
Jul 08 07:05:30.591538 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.591509    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dfc67dd-d6b3-494e-b690-c2c17818c048" containerName="authorino"
Jul 08 07:05:30.595644 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.595626    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-5bxlp"
Jul 08 07:05:30.599041 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.599016    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cxt5w\""
Jul 08 07:05:30.599041 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.599031    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 08 07:05:30.601737 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.601709    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-5bxlp"]
Jul 08 07:05:30.742101 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.742076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b975a5c2-cb2f-4047-bd0b-a50ef9456b10-tls-cert\") pod \"authorino-7c77957575-5bxlp\" (UID: \"b975a5c2-cb2f-4047-bd0b-a50ef9456b10\") " pod="kuadrant-system/authorino-7c77957575-5bxlp"
Jul 08 07:05:30.742259 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.742113    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwnvk\" (UniqueName: \"kubernetes.io/projected/b975a5c2-cb2f-4047-bd0b-a50ef9456b10-kube-api-access-dwnvk\") pod \"authorino-7c77957575-5bxlp\" (UID: \"b975a5c2-cb2f-4047-bd0b-a50ef9456b10\") " pod="kuadrant-system/authorino-7c77957575-5bxlp"
Jul 08 07:05:30.843062 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.842996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b975a5c2-cb2f-4047-bd0b-a50ef9456b10-tls-cert\") pod \"authorino-7c77957575-5bxlp\" (UID: \"b975a5c2-cb2f-4047-bd0b-a50ef9456b10\") " pod="kuadrant-system/authorino-7c77957575-5bxlp"
Jul 08 07:05:30.843062 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.843034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwnvk\" (UniqueName: \"kubernetes.io/projected/b975a5c2-cb2f-4047-bd0b-a50ef9456b10-kube-api-access-dwnvk\") pod \"authorino-7c77957575-5bxlp\" (UID: \"b975a5c2-cb2f-4047-bd0b-a50ef9456b10\") " pod="kuadrant-system/authorino-7c77957575-5bxlp"
Jul 08 07:05:30.845371 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.845345    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b975a5c2-cb2f-4047-bd0b-a50ef9456b10-tls-cert\") pod \"authorino-7c77957575-5bxlp\" (UID: \"b975a5c2-cb2f-4047-bd0b-a50ef9456b10\") " pod="kuadrant-system/authorino-7c77957575-5bxlp"
Jul 08 07:05:30.851827 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.851801    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwnvk\" (UniqueName: \"kubernetes.io/projected/b975a5c2-cb2f-4047-bd0b-a50ef9456b10-kube-api-access-dwnvk\") pod \"authorino-7c77957575-5bxlp\" (UID: \"b975a5c2-cb2f-4047-bd0b-a50ef9456b10\") " pod="kuadrant-system/authorino-7c77957575-5bxlp"
Jul 08 07:05:30.904983 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:30.904963    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-5bxlp"
Jul 08 07:05:31.026352 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:31.026318    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-5bxlp"]
Jul 08 07:05:31.029685 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:05:31.029660    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb975a5c2_cb2f_4047_bd0b_a50ef9456b10.slice/crio-caa4240fb1ac3ef6e1e24ff5c2273c4d1981791d570325110e018e43350cec1c WatchSource:0}: Error finding container caa4240fb1ac3ef6e1e24ff5c2273c4d1981791d570325110e018e43350cec1c: Status 404 returned error can't find the container with id caa4240fb1ac3ef6e1e24ff5c2273c4d1981791d570325110e018e43350cec1c
Jul 08 07:05:31.077465 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:31.077431    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-5bxlp" event={"ID":"b975a5c2-cb2f-4047-bd0b-a50ef9456b10","Type":"ContainerStarted","Data":"caa4240fb1ac3ef6e1e24ff5c2273c4d1981791d570325110e018e43350cec1c"}
Jul 08 07:05:32.082532 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:32.082497    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-5bxlp" event={"ID":"b975a5c2-cb2f-4047-bd0b-a50ef9456b10","Type":"ContainerStarted","Data":"094697c6367727079f6862d2a95edee4595d06818c7d891824a576ced4008a91"}
Jul 08 07:05:32.098596 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:32.098542    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-5bxlp" podStartSLOduration=1.750502387 podStartE2EDuration="2.098525088s" totalImagesPullingTime="348.022701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:05:32.096975875 +0000 UTC m=+647.900762288" watchObservedRunningTime="2026-07-08 07:05:32.098525088 +0000 UTC m=+647.902311488"
Jul 08 07:05:34.928327 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:34.928299    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:05:36.288771 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.288739    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-9487895bf-nvzts"]
Jul 08 07:05:36.289265 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.288913    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-9487895bf-nvzts" podUID="812e7060-1958-4e43-8bc7-3c163bd8724e" containerName="manager" containerID="cri-o://c487b866b70ab7dd5c998291714b7d51a62ed01ef93e74917dd14b69eec01f1b" gracePeriod=10
Jul 08 07:05:36.311931 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.311902    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-9487895bf-gwpxw"]
Jul 08 07:05:36.315305 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.315287    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9487895bf-gwpxw"
Jul 08 07:05:36.323211 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.323188    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-9487895bf-gwpxw"]
Jul 08 07:05:36.481658 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.481617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvc9\" (UniqueName: \"kubernetes.io/projected/1bbd95a9-924b-43e2-9803-f0bc0d3d899c-kube-api-access-2pvc9\") pod \"kserve-controller-manager-9487895bf-gwpxw\" (UID: \"1bbd95a9-924b-43e2-9803-f0bc0d3d899c\") " pod="kserve/kserve-controller-manager-9487895bf-gwpxw"
Jul 08 07:05:36.481801 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.481751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bbd95a9-924b-43e2-9803-f0bc0d3d899c-cert\") pod \"kserve-controller-manager-9487895bf-gwpxw\" (UID: \"1bbd95a9-924b-43e2-9803-f0bc0d3d899c\") " pod="kserve/kserve-controller-manager-9487895bf-gwpxw"
Jul 08 07:05:36.526326 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.526301    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9487895bf-nvzts"
Jul 08 07:05:36.582422 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.582358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bbd95a9-924b-43e2-9803-f0bc0d3d899c-cert\") pod \"kserve-controller-manager-9487895bf-gwpxw\" (UID: \"1bbd95a9-924b-43e2-9803-f0bc0d3d899c\") " pod="kserve/kserve-controller-manager-9487895bf-gwpxw"
Jul 08 07:05:36.582550 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.582514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvc9\" (UniqueName: \"kubernetes.io/projected/1bbd95a9-924b-43e2-9803-f0bc0d3d899c-kube-api-access-2pvc9\") pod \"kserve-controller-manager-9487895bf-gwpxw\" (UID: \"1bbd95a9-924b-43e2-9803-f0bc0d3d899c\") " pod="kserve/kserve-controller-manager-9487895bf-gwpxw"
Jul 08 07:05:36.584750 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.584721    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bbd95a9-924b-43e2-9803-f0bc0d3d899c-cert\") pod \"kserve-controller-manager-9487895bf-gwpxw\" (UID: \"1bbd95a9-924b-43e2-9803-f0bc0d3d899c\") " pod="kserve/kserve-controller-manager-9487895bf-gwpxw"
Jul 08 07:05:36.590531 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.590511    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvc9\" (UniqueName: \"kubernetes.io/projected/1bbd95a9-924b-43e2-9803-f0bc0d3d899c-kube-api-access-2pvc9\") pod \"kserve-controller-manager-9487895bf-gwpxw\" (UID: \"1bbd95a9-924b-43e2-9803-f0bc0d3d899c\") " pod="kserve/kserve-controller-manager-9487895bf-gwpxw"
Jul 08 07:05:36.667425 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.667403    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9487895bf-gwpxw"
Jul 08 07:05:36.682911 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.682891    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/812e7060-1958-4e43-8bc7-3c163bd8724e-cert\") pod \"812e7060-1958-4e43-8bc7-3c163bd8724e\" (UID: \"812e7060-1958-4e43-8bc7-3c163bd8724e\") "
Jul 08 07:05:36.683008 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.682943    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp5q2\" (UniqueName: \"kubernetes.io/projected/812e7060-1958-4e43-8bc7-3c163bd8724e-kube-api-access-mp5q2\") pod \"812e7060-1958-4e43-8bc7-3c163bd8724e\" (UID: \"812e7060-1958-4e43-8bc7-3c163bd8724e\") "
Jul 08 07:05:36.684811 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.684785    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/812e7060-1958-4e43-8bc7-3c163bd8724e-cert" (OuterVolumeSpecName: "cert") pod "812e7060-1958-4e43-8bc7-3c163bd8724e" (UID: "812e7060-1958-4e43-8bc7-3c163bd8724e"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:05:36.684935 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.684908    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812e7060-1958-4e43-8bc7-3c163bd8724e-kube-api-access-mp5q2" (OuterVolumeSpecName: "kube-api-access-mp5q2") pod "812e7060-1958-4e43-8bc7-3c163bd8724e" (UID: "812e7060-1958-4e43-8bc7-3c163bd8724e"). InnerVolumeSpecName "kube-api-access-mp5q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:05:36.783684 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.783647    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp5q2\" (UniqueName: \"kubernetes.io/projected/812e7060-1958-4e43-8bc7-3c163bd8724e-kube-api-access-mp5q2\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:05:36.783684 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.783680    2582 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/812e7060-1958-4e43-8bc7-3c163bd8724e-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:05:36.810728 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:36.810698    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-9487895bf-gwpxw"]
Jul 08 07:05:36.813705 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:05:36.813671    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bbd95a9_924b_43e2_9803_f0bc0d3d899c.slice/crio-df989063f8334af1e60ad1369dcdb3c03693d511edc0f46ae9ffb60da78f5b96 WatchSource:0}: Error finding container df989063f8334af1e60ad1369dcdb3c03693d511edc0f46ae9ffb60da78f5b96: Status 404 returned error can't find the container with id df989063f8334af1e60ad1369dcdb3c03693d511edc0f46ae9ffb60da78f5b96
Jul 08 07:05:37.101487 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:37.101456    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9487895bf-gwpxw" event={"ID":"1bbd95a9-924b-43e2-9803-f0bc0d3d899c","Type":"ContainerStarted","Data":"df989063f8334af1e60ad1369dcdb3c03693d511edc0f46ae9ffb60da78f5b96"}
Jul 08 07:05:37.102581 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:37.102558    2582 generic.go:356] "Generic (PLEG): container finished" podID="812e7060-1958-4e43-8bc7-3c163bd8724e" containerID="c487b866b70ab7dd5c998291714b7d51a62ed01ef93e74917dd14b69eec01f1b" exitCode=0
Jul 08 07:05:37.102688 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:37.102603    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9487895bf-nvzts"
Jul 08 07:05:37.102688 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:37.102609    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9487895bf-nvzts" event={"ID":"812e7060-1958-4e43-8bc7-3c163bd8724e","Type":"ContainerDied","Data":"c487b866b70ab7dd5c998291714b7d51a62ed01ef93e74917dd14b69eec01f1b"}
Jul 08 07:05:37.102688 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:37.102643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9487895bf-nvzts" event={"ID":"812e7060-1958-4e43-8bc7-3c163bd8724e","Type":"ContainerDied","Data":"76b21fd2b0f85d8908d39c4ba00b4ab6eb994f79d46549437373ee42dac0a0b2"}
Jul 08 07:05:37.102688 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:37.102667    2582 scope.go:117] "RemoveContainer" containerID="c487b866b70ab7dd5c998291714b7d51a62ed01ef93e74917dd14b69eec01f1b"
Jul 08 07:05:37.110043 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:37.110026    2582 scope.go:117] "RemoveContainer" containerID="c487b866b70ab7dd5c998291714b7d51a62ed01ef93e74917dd14b69eec01f1b"
Jul 08 07:05:37.110324 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:05:37.110306    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c487b866b70ab7dd5c998291714b7d51a62ed01ef93e74917dd14b69eec01f1b\": container with ID starting with c487b866b70ab7dd5c998291714b7d51a62ed01ef93e74917dd14b69eec01f1b not found: ID does not exist" containerID="c487b866b70ab7dd5c998291714b7d51a62ed01ef93e74917dd14b69eec01f1b"
Jul 08 07:05:37.110400 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:37.110329    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c487b866b70ab7dd5c998291714b7d51a62ed01ef93e74917dd14b69eec01f1b"} err="failed to get container status \"c487b866b70ab7dd5c998291714b7d51a62ed01ef93e74917dd14b69eec01f1b\": rpc error: code = NotFound desc = could not find container \"c487b866b70ab7dd5c998291714b7d51a62ed01ef93e74917dd14b69eec01f1b\": container with ID starting with c487b866b70ab7dd5c998291714b7d51a62ed01ef93e74917dd14b69eec01f1b not found: ID does not exist"
Jul 08 07:05:37.118513 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:37.118494    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-9487895bf-nvzts"]
Jul 08 07:05:37.122590 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:37.122571    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-9487895bf-nvzts"]
Jul 08 07:05:38.108655 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:38.108618    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9487895bf-gwpxw" event={"ID":"1bbd95a9-924b-43e2-9803-f0bc0d3d899c","Type":"ContainerStarted","Data":"0880ba0ce8f23d619c4362c99cbad1f83ee7343d5381a1c51878c062af97fe16"}
Jul 08 07:05:38.109024 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:38.108749    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-9487895bf-gwpxw"
Jul 08 07:05:38.126013 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:38.125970    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-9487895bf-gwpxw" podStartSLOduration=1.76027176 podStartE2EDuration="2.125958046s" totalImagesPullingTime="365.686286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:05:38.124138097 +0000 UTC m=+653.927924516" watchObservedRunningTime="2026-07-08 07:05:38.125958046 +0000 UTC m=+653.929744445"
Jul 08 07:05:38.745607 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:38.745576    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="812e7060-1958-4e43-8bc7-3c163bd8724e" path="/var/lib/kubelet/pods/812e7060-1958-4e43-8bc7-3c163bd8724e/volumes"
Jul 08 07:05:51.135693 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:51.135652    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:05:51.197966 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:05:51.197938    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:06:09.118371 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:06:09.118345    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-9487895bf-gwpxw"
Jul 08 07:06:10.113905 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:06:10.113878    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:06:10.243328 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:06:10.243297    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:06:10.343589 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:06:10.343559    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:06:10.452295 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:06:10.452268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:06:10.759345 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:06:10.759266    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:06:10.823635 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:06:10.823599    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:07:45.848086 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:07:45.848055    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:07:45.973258 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:07:45.973225    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:07:58.275088 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:07:58.275054    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:07:58.384484 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:07:58.384455    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:08:02.193493 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:02.193464    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:08:07.580466 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.580389    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"]
Jul 08 07:08:07.580811 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.580713    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="812e7060-1958-4e43-8bc7-3c163bd8724e" containerName="manager"
Jul 08 07:08:07.580811 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.580726    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="812e7060-1958-4e43-8bc7-3c163bd8724e" containerName="manager"
Jul 08 07:08:07.580811 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.580782    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="812e7060-1958-4e43-8bc7-3c163bd8724e" containerName="manager"
Jul 08 07:08:07.583638 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.583619    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.590136 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.590116    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 07:08:07.591471 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.591448    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 08 07:08:07.591583 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.591478    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 07:08:07.591583 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.591497    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-psjvw\""
Jul 08 07:08:07.602079 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.602058    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"]
Jul 08 07:08:07.627297 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.627277    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:08:07.639346 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.639329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.639422 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.639361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.639422 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.639387    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.639515 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.639448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275044ab-39d1-4cde-81b1-1fd9271ddd49-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.639515 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.639481    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqzgv\" (UniqueName: \"kubernetes.io/projected/275044ab-39d1-4cde-81b1-1fd9271ddd49-kube-api-access-dqzgv\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.639612 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.639562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.740188 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.740139    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.740348 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.740196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.740348 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.740226    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275044ab-39d1-4cde-81b1-1fd9271ddd49-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.740348 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.740248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqzgv\" (UniqueName: \"kubernetes.io/projected/275044ab-39d1-4cde-81b1-1fd9271ddd49-kube-api-access-dqzgv\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.740348 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.740273    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.740348 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.740309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.740608 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.740538    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.740663 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.740605    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.740709 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.740660    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.742617 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.742584    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.742716 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.742681    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275044ab-39d1-4cde-81b1-1fd9271ddd49-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.750002 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.749978    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqzgv\" (UniqueName: \"kubernetes.io/projected/275044ab-39d1-4cde-81b1-1fd9271ddd49-kube-api-access-dqzgv\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:07.893225 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:07.893117    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:08.033992 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:08.033957    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"]
Jul 08 07:08:08.036926 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:08:08.036898    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275044ab_39d1_4cde_81b1_1fd9271ddd49.slice/crio-5b912e6f97dd282037e04722ad55386c581f5cd13afd63ec5d1d27d3684988e7 WatchSource:0}: Error finding container 5b912e6f97dd282037e04722ad55386c581f5cd13afd63ec5d1d27d3684988e7: Status 404 returned error can't find the container with id 5b912e6f97dd282037e04722ad55386c581f5cd13afd63ec5d1d27d3684988e7
Jul 08 07:08:08.038893 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:08.038875    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 07:08:08.701823 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:08.701747    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf" event={"ID":"275044ab-39d1-4cde-81b1-1fd9271ddd49","Type":"ContainerStarted","Data":"5b912e6f97dd282037e04722ad55386c581f5cd13afd63ec5d1d27d3684988e7"}
Jul 08 07:08:09.707813 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:09.707767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf" event={"ID":"275044ab-39d1-4cde-81b1-1fd9271ddd49","Type":"ContainerStarted","Data":"da2584e3a341e23e16e60d15242aec1cf427a04802c1b0dec0124d052084441d"}
Jul 08 07:08:09.724994 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:09.724946    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf" podStartSLOduration=1.492113683 podStartE2EDuration="2.724930086s" totalImagesPullingTime="1.232816403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:08:09.723714944 +0000 UTC m=+805.527501377" watchObservedRunningTime="2026-07-08 07:08:09.724930086 +0000 UTC m=+805.528716485"
Jul 08 07:08:09.893988 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:09.893964    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:12.510440 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:12.510404    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:08:13.670960 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:13.670923    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:08:16.262718 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:16.262683    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"]
Jul 08 07:08:16.274320 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:16.274292    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:08:17.847207 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:17.847146    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:08:18.655383 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:18.655347    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:08:24.788824 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:24.788786    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerID="36c3a6e0cf7013208444f6739eb265949fe41b00dd4324d500c56c2c225d2206" exitCode=0
Jul 08 07:08:24.789250 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:24.788827    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerDied","Data":"36c3a6e0cf7013208444f6739eb265949fe41b00dd4324d500c56c2c225d2206"}
Jul 08 07:08:24.789250 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:24.789188    2582 scope.go:117] "RemoveContainer" containerID="36c3a6e0cf7013208444f6739eb265949fe41b00dd4324d500c56c2c225d2206"
Jul 08 07:08:24.790136 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:24.790105    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:08:24.790256 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:24.790182    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:08:24.790256 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:24.790213    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:08:25.795559 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:25.795523    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerStarted","Data":"54ea71c8c9928bc406ca46baa093eb0032910d5a011465310c29efeb9e31882b"}
Jul 08 07:08:26.306007 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:26.305968    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:08:26.515318 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:26.515283    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:08:26.515488 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:26.515464    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:08:26.515538 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:26.515514    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:08:27.515578 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:27.515544    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:08:27.515967 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:27.515605    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:08:28.515818 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:28.515784    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:08:28.516246 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:28.515840    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:08:29.372776 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:29.372745    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:08:29.515741 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:29.515700    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:08:29.515941 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:29.515756    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:08:30.514908 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:30.514872    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:08:30.515115 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:30.514936    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:08:31.285983 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.285931    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf" podUID="275044ab-39d1-4cde-81b1-1fd9271ddd49" containerName="main" containerID="cri-o://da2584e3a341e23e16e60d15242aec1cf427a04802c1b0dec0124d052084441d" gracePeriod=45
Jul 08 07:08:31.407409 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.407378    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:08:31.522478 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.522451    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:31.626626 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.626514    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-home\") pod \"275044ab-39d1-4cde-81b1-1fd9271ddd49\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") "
Jul 08 07:08:31.626794 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.626634    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqzgv\" (UniqueName: \"kubernetes.io/projected/275044ab-39d1-4cde-81b1-1fd9271ddd49-kube-api-access-dqzgv\") pod \"275044ab-39d1-4cde-81b1-1fd9271ddd49\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") "
Jul 08 07:08:31.626794 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.626676    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-model-cache\") pod \"275044ab-39d1-4cde-81b1-1fd9271ddd49\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") "
Jul 08 07:08:31.626794 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.626732    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275044ab-39d1-4cde-81b1-1fd9271ddd49-tls-certs\") pod \"275044ab-39d1-4cde-81b1-1fd9271ddd49\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") "
Jul 08 07:08:31.626794 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.626770    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-tmp-dir\") pod \"275044ab-39d1-4cde-81b1-1fd9271ddd49\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") "
Jul 08 07:08:31.627002 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.626809    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-dshm\") pod \"275044ab-39d1-4cde-81b1-1fd9271ddd49\" (UID: \"275044ab-39d1-4cde-81b1-1fd9271ddd49\") "
Jul 08 07:08:31.627002 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.626878    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-home" (OuterVolumeSpecName: "home") pod "275044ab-39d1-4cde-81b1-1fd9271ddd49" (UID: "275044ab-39d1-4cde-81b1-1fd9271ddd49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:08:31.627111 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.627094    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:08:31.627247 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.627208    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-model-cache" (OuterVolumeSpecName: "model-cache") pod "275044ab-39d1-4cde-81b1-1fd9271ddd49" (UID: "275044ab-39d1-4cde-81b1-1fd9271ddd49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:08:31.627398 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.627358    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "275044ab-39d1-4cde-81b1-1fd9271ddd49" (UID: "275044ab-39d1-4cde-81b1-1fd9271ddd49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:08:31.629086 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.629054    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275044ab-39d1-4cde-81b1-1fd9271ddd49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "275044ab-39d1-4cde-81b1-1fd9271ddd49" (UID: "275044ab-39d1-4cde-81b1-1fd9271ddd49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:08:31.629263 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.629246    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-dshm" (OuterVolumeSpecName: "dshm") pod "275044ab-39d1-4cde-81b1-1fd9271ddd49" (UID: "275044ab-39d1-4cde-81b1-1fd9271ddd49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:08:31.629403 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.629380    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275044ab-39d1-4cde-81b1-1fd9271ddd49-kube-api-access-dqzgv" (OuterVolumeSpecName: "kube-api-access-dqzgv") pod "275044ab-39d1-4cde-81b1-1fd9271ddd49" (UID: "275044ab-39d1-4cde-81b1-1fd9271ddd49"). InnerVolumeSpecName "kube-api-access-dqzgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:08:31.635216 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.635195    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:08:31.635810 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.635789    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:08:31.636451 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.636430    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:08:31.727680 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.727646    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqzgv\" (UniqueName: \"kubernetes.io/projected/275044ab-39d1-4cde-81b1-1fd9271ddd49-kube-api-access-dqzgv\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:08:31.727680 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.727677    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:08:31.727887 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.727699    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275044ab-39d1-4cde-81b1-1fd9271ddd49-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:08:31.727887 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.727710    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:08:31.727887 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.727722    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275044ab-39d1-4cde-81b1-1fd9271ddd49-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:08:31.820344 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.820317    2582 generic.go:356] "Generic (PLEG): container finished" podID="275044ab-39d1-4cde-81b1-1fd9271ddd49" containerID="da2584e3a341e23e16e60d15242aec1cf427a04802c1b0dec0124d052084441d" exitCode=0
Jul 08 07:08:31.820459 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.820363    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"
Jul 08 07:08:31.820459 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.820411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf" event={"ID":"275044ab-39d1-4cde-81b1-1fd9271ddd49","Type":"ContainerDied","Data":"da2584e3a341e23e16e60d15242aec1cf427a04802c1b0dec0124d052084441d"}
Jul 08 07:08:31.820571 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.820463    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf" event={"ID":"275044ab-39d1-4cde-81b1-1fd9271ddd49","Type":"ContainerDied","Data":"5b912e6f97dd282037e04722ad55386c581f5cd13afd63ec5d1d27d3684988e7"}
Jul 08 07:08:31.820571 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.820493    2582 scope.go:117] "RemoveContainer" containerID="da2584e3a341e23e16e60d15242aec1cf427a04802c1b0dec0124d052084441d"
Jul 08 07:08:31.828570 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.828555    2582 scope.go:117] "RemoveContainer" containerID="da2584e3a341e23e16e60d15242aec1cf427a04802c1b0dec0124d052084441d"
Jul 08 07:08:31.828824 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:08:31.828808    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2584e3a341e23e16e60d15242aec1cf427a04802c1b0dec0124d052084441d\": container with ID starting with da2584e3a341e23e16e60d15242aec1cf427a04802c1b0dec0124d052084441d not found: ID does not exist" containerID="da2584e3a341e23e16e60d15242aec1cf427a04802c1b0dec0124d052084441d"
Jul 08 07:08:31.828879 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.828830    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2584e3a341e23e16e60d15242aec1cf427a04802c1b0dec0124d052084441d"} err="failed to get container status \"da2584e3a341e23e16e60d15242aec1cf427a04802c1b0dec0124d052084441d\": rpc error: code = NotFound desc = could not find container \"da2584e3a341e23e16e60d15242aec1cf427a04802c1b0dec0124d052084441d\": container with ID starting with da2584e3a341e23e16e60d15242aec1cf427a04802c1b0dec0124d052084441d not found: ID does not exist"
Jul 08 07:08:31.841212 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.841193    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"]
Jul 08 07:08:31.844514 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:31.844494    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5plmgf"]
Jul 08 07:08:32.745985 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:32.745950    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="275044ab-39d1-4cde-81b1-1fd9271ddd49" path="/var/lib/kubelet/pods/275044ab-39d1-4cde-81b1-1fd9271ddd49/volumes"
Jul 08 07:08:32.913047 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:32.913012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:08:42.395286 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.395252    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"]
Jul 08 07:08:42.395661 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.395522    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="275044ab-39d1-4cde-81b1-1fd9271ddd49" containerName="main"
Jul 08 07:08:42.395661 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.395533    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="275044ab-39d1-4cde-81b1-1fd9271ddd49" containerName="main"
Jul 08 07:08:42.395661 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.395587    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="275044ab-39d1-4cde-81b1-1fd9271ddd49" containerName="main"
Jul 08 07:08:42.398290 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.398273    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.401053 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.401031    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 07:08:42.401204 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.401142    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-psjvw\""
Jul 08 07:08:42.401204 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.401185    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 08 07:08:42.401328 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.401214    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 07:08:42.401328 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.401255    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 07:08:42.408306 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.408270    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"]
Jul 08 07:08:42.475918 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.475882    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:08:42.500926 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.500903    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-tmp-dir\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.501050 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.500934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhgz\" (UniqueName: \"kubernetes.io/projected/57f5b1d9-b254-4410-8937-06a6da16c06a-kube-api-access-gvhgz\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.501050 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.500961    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-dshm\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.501050 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.501030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-model-cache\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.501263 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.501073    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f5b1d9-b254-4410-8937-06a6da16c06a-tls-certs\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.501263 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.501095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-kserve-provision-location\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.501263 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.501129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-home\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.602194 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.602163    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-tmp-dir\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.602338 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.602201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhgz\" (UniqueName: \"kubernetes.io/projected/57f5b1d9-b254-4410-8937-06a6da16c06a-kube-api-access-gvhgz\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.602338 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.602312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-dshm\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.602469 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.602344    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-model-cache\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.602469 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.602378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f5b1d9-b254-4410-8937-06a6da16c06a-tls-certs\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.602469 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.602403    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-kserve-provision-location\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.602469 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.602449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-home\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.602674 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.602611    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-tmp-dir\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.602736 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.602713    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-model-cache\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.602794 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.602752    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-home\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.602838 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.602802    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-kserve-provision-location\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.604545 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.604525    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-dshm\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.604848 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.604833    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f5b1d9-b254-4410-8937-06a6da16c06a-tls-certs\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.610122 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.610097    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhgz\" (UniqueName: \"kubernetes.io/projected/57f5b1d9-b254-4410-8937-06a6da16c06a-kube-api-access-gvhgz\") pod \"auth-enabled-test-kserve-b46fb9c55-wbzwg\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.709030 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.709000    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:08:42.732296 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.732075    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"]
Jul 08 07:08:42.736490 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.736472    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:42.739783 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.739759    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-epp-sa-dockercfg-6s6sp\""
Jul 08 07:08:42.749859 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.749708    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"]
Jul 08 07:08:42.769571 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.769540    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:08:42.803813 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.803778    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776c7f7b-85ca-438f-a932-3875fcf4e653-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:42.803931 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.803897    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/776c7f7b-85ca-438f-a932-3875fcf4e653-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:42.803931 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.803940    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jxfn\" (UniqueName: \"kubernetes.io/projected/776c7f7b-85ca-438f-a932-3875fcf4e653-kube-api-access-6jxfn\") pod \"auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:42.804129 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.804007    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/776c7f7b-85ca-438f-a932-3875fcf4e653-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:42.852246 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.852224    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"]
Jul 08 07:08:42.854082 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:08:42.854055    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f5b1d9_b254_4410_8937_06a6da16c06a.slice/crio-b64050df225ceff9b79da2a2514b09a7d3e4c9d10fa2cf657c767a66793fc4b5 WatchSource:0}: Error finding container b64050df225ceff9b79da2a2514b09a7d3e4c9d10fa2cf657c767a66793fc4b5: Status 404 returned error can't find the container with id b64050df225ceff9b79da2a2514b09a7d3e4c9d10fa2cf657c767a66793fc4b5
Jul 08 07:08:42.864726 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.864698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" event={"ID":"57f5b1d9-b254-4410-8937-06a6da16c06a","Type":"ContainerStarted","Data":"b64050df225ceff9b79da2a2514b09a7d3e4c9d10fa2cf657c767a66793fc4b5"}
Jul 08 07:08:42.904623 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.904597    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776c7f7b-85ca-438f-a932-3875fcf4e653-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:42.904723 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.904675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/776c7f7b-85ca-438f-a932-3875fcf4e653-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:42.904723 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.904708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxfn\" (UniqueName: \"kubernetes.io/projected/776c7f7b-85ca-438f-a932-3875fcf4e653-kube-api-access-6jxfn\") pod \"auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:42.904831 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.904742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/776c7f7b-85ca-438f-a932-3875fcf4e653-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:42.905076 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.905059    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/776c7f7b-85ca-438f-a932-3875fcf4e653-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:42.905145 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.905118    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/776c7f7b-85ca-438f-a932-3875fcf4e653-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:42.906828 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.906808    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776c7f7b-85ca-438f-a932-3875fcf4e653-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:42.912837 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:42.912815    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jxfn\" (UniqueName: \"kubernetes.io/projected/776c7f7b-85ca-438f-a932-3875fcf4e653-kube-api-access-6jxfn\") pod \"auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:43.047625 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:43.047570    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:43.175056 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:43.174984    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"]
Jul 08 07:08:43.177089 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:08:43.177061    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776c7f7b_85ca_438f_a932_3875fcf4e653.slice/crio-e3d01b12de172af334df8cfe0c1ac630c935ca99242ffd49aad289483407f73f WatchSource:0}: Error finding container e3d01b12de172af334df8cfe0c1ac630c935ca99242ffd49aad289483407f73f: Status 404 returned error can't find the container with id e3d01b12de172af334df8cfe0c1ac630c935ca99242ffd49aad289483407f73f
Jul 08 07:08:43.871996 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:43.871949    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t" event={"ID":"776c7f7b-85ca-438f-a932-3875fcf4e653","Type":"ContainerStarted","Data":"e3d01b12de172af334df8cfe0c1ac630c935ca99242ffd49aad289483407f73f"}
Jul 08 07:08:46.205972 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:46.205945    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 07:08:46.886960 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:46.886921    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" event={"ID":"57f5b1d9-b254-4410-8937-06a6da16c06a","Type":"ContainerStarted","Data":"c47d6f51c9884954d93aad88bbe02695cba80233e6ed56c9ef83db18a19bee2e"}
Jul 08 07:08:46.888401 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:46.888374    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t" event={"ID":"776c7f7b-85ca-438f-a932-3875fcf4e653","Type":"ContainerStarted","Data":"4598cb7fb05be881057c3f092cf2354faa5a2e823bc3f921607f39a45b82fe82"}
Jul 08 07:08:47.894101 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:47.894069    2582 generic.go:356] "Generic (PLEG): container finished" podID="776c7f7b-85ca-438f-a932-3875fcf4e653" containerID="4598cb7fb05be881057c3f092cf2354faa5a2e823bc3f921607f39a45b82fe82" exitCode=0
Jul 08 07:08:47.894519 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:47.894170    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t" event={"ID":"776c7f7b-85ca-438f-a932-3875fcf4e653","Type":"ContainerDied","Data":"4598cb7fb05be881057c3f092cf2354faa5a2e823bc3f921607f39a45b82fe82"}
Jul 08 07:08:49.904095 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:49.904064    2582 generic.go:356] "Generic (PLEG): container finished" podID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerID="c47d6f51c9884954d93aad88bbe02695cba80233e6ed56c9ef83db18a19bee2e" exitCode=0
Jul 08 07:08:49.904470 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:49.904133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" event={"ID":"57f5b1d9-b254-4410-8937-06a6da16c06a","Type":"ContainerDied","Data":"c47d6f51c9884954d93aad88bbe02695cba80233e6ed56c9ef83db18a19bee2e"}
Jul 08 07:08:49.906008 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:49.905991    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t" event={"ID":"776c7f7b-85ca-438f-a932-3875fcf4e653","Type":"ContainerStarted","Data":"198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155"}
Jul 08 07:08:49.906137 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:49.906120    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:08:49.941976 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:08:49.941933    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t" podStartSLOduration=2.497928708 podStartE2EDuration="7.941915557s" totalImagesPullingTime="4.732324582s" totalInitContainerRuntime="711.662267ms" isStatefulPod=false podCreationTimestamp="2026-07-08 07:08:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:08:49.941591054 +0000 UTC m=+845.745377454" watchObservedRunningTime="2026-07-08 07:08:49.941915557 +0000 UTC m=+845.745701958"
Jul 08 07:09:02.112959 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:09:02.112923    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:09:20.915821 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:09:20.915794    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:09:26.337508 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:09:26.337381    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:09:26.396670 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:09:26.396637    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:09:37.142497 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:09:37.142403    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" event={"ID":"57f5b1d9-b254-4410-8937-06a6da16c06a","Type":"ContainerStarted","Data":"f32eb32767656d445093f5981c4f9399dcdccc0a98a6acf8033845f41e54943c"}
Jul 08 07:09:37.165867 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:09:37.165819    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" podStartSLOduration=1.7832406079999998 podStartE2EDuration="55.165801867s" totalImagesPullingTime="50.262734734s" totalInitContainerRuntime="3.119826525s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:08:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:09:37.163881309 +0000 UTC m=+892.967667709" watchObservedRunningTime="2026-07-08 07:09:37.165801867 +0000 UTC m=+892.969588267"
Jul 08 07:09:37.710265 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:09:37.710232    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:09:42.709624 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:09:42.709575    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:09:42.711237 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:09:42.711207    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 08 07:09:44.661004 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:09:44.660974    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:09:44.664846 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:09:44.664826    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:09:52.710248 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:09:52.710202    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 08 07:10:02.710162 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:10:02.710117    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 08 07:10:05.979616 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:10:05.979509    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:10:07.430826 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:10:07.430791    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:10:08.167709 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:10:08.167649    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:10:12.709718 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:10:12.709678    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 08 07:10:18.500837 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:10:18.500800    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:10:21.784842 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:10:21.784808    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:10:21.884798 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:10:21.884768    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:10:22.709745 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:10:22.709707    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 08 07:10:32.709950 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:10:32.709905    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 08 07:10:40.417879 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:10:40.417844    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:10:40.480902 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:10:40.480867    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:10:42.709780 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:10:42.709725    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 08 07:10:52.709662 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:10:52.709623    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 08 07:11:02.710542 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:02.710495    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 08 07:11:07.718163 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:07.718073    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:11:07.960015 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:07.959975    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:11:12.710367 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:12.710331    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 08 07:11:22.709800 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:22.709762    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Jul 08 07:11:24.731204 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:24.731127    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:11:32.719715 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:32.719679    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:11:32.727325 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:32.727302    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:11:37.852591 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:37.852558    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"]
Jul 08 07:11:37.859894 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:37.859863    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:11:37.865572 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:37.865548    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"]
Jul 08 07:11:37.873623 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:11:37.873573    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 07:11:37.873623 ip-10-0-140-110 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 07:11:37.873623 ip-10-0-140-110 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 08 07:11:37.873623 ip-10-0-140-110 kubenswrapper[2582]:  > containerID="198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155" cmd=["/bin/sleep","15"]
Jul 08 07:11:37.873855 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:37.873651    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t" message="" err=<
Jul 08 07:11:37.873855 ip-10-0-140-110 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 07:11:37.873855 ip-10-0-140-110 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 08 07:11:37.873855 ip-10-0-140-110 kubenswrapper[2582]:  >
Jul 08 07:11:37.873855 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:11:37.873670    2582 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 07:11:37.873855 ip-10-0-140-110 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 07:11:37.873855 ip-10-0-140-110 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 08 07:11:37.873855 ip-10-0-140-110 kubenswrapper[2582]:  > pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t" podUID="776c7f7b-85ca-438f-a932-3875fcf4e653" containerName="main" containerID="cri-o://198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155"
Jul 08 07:11:37.873855 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:37.873727    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t" podUID="776c7f7b-85ca-438f-a932-3875fcf4e653" containerName="main" containerID="cri-o://198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155" gracePeriod=60
Jul 08 07:11:37.935102 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:37.935082    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:11:38.130722 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.130700    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:11:38.201823 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.201802    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776c7f7b-85ca-438f-a932-3875fcf4e653-tls-certs\") pod \"776c7f7b-85ca-438f-a932-3875fcf4e653\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") "
Jul 08 07:11:38.201936 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.201844    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jxfn\" (UniqueName: \"kubernetes.io/projected/776c7f7b-85ca-438f-a932-3875fcf4e653-kube-api-access-6jxfn\") pod \"776c7f7b-85ca-438f-a932-3875fcf4e653\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") "
Jul 08 07:11:38.201936 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.201879    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/776c7f7b-85ca-438f-a932-3875fcf4e653-kserve-provision-location\") pod \"776c7f7b-85ca-438f-a932-3875fcf4e653\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") "
Jul 08 07:11:38.202007 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.201932    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/776c7f7b-85ca-438f-a932-3875fcf4e653-tokenizer-uds\") pod \"776c7f7b-85ca-438f-a932-3875fcf4e653\" (UID: \"776c7f7b-85ca-438f-a932-3875fcf4e653\") "
Jul 08 07:11:38.202269 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.202240    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776c7f7b-85ca-438f-a932-3875fcf4e653-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "776c7f7b-85ca-438f-a932-3875fcf4e653" (UID: "776c7f7b-85ca-438f-a932-3875fcf4e653"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:11:38.202624 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.202601    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776c7f7b-85ca-438f-a932-3875fcf4e653-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "776c7f7b-85ca-438f-a932-3875fcf4e653" (UID: "776c7f7b-85ca-438f-a932-3875fcf4e653"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:11:38.203898 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.203874    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776c7f7b-85ca-438f-a932-3875fcf4e653-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "776c7f7b-85ca-438f-a932-3875fcf4e653" (UID: "776c7f7b-85ca-438f-a932-3875fcf4e653"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:11:38.203980 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.203909    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776c7f7b-85ca-438f-a932-3875fcf4e653-kube-api-access-6jxfn" (OuterVolumeSpecName: "kube-api-access-6jxfn") pod "776c7f7b-85ca-438f-a932-3875fcf4e653" (UID: "776c7f7b-85ca-438f-a932-3875fcf4e653"). InnerVolumeSpecName "kube-api-access-6jxfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:11:38.303051 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.303017    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jxfn\" (UniqueName: \"kubernetes.io/projected/776c7f7b-85ca-438f-a932-3875fcf4e653-kube-api-access-6jxfn\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:11:38.303051 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.303050    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/776c7f7b-85ca-438f-a932-3875fcf4e653-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:11:38.303051 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.303063    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/776c7f7b-85ca-438f-a932-3875fcf4e653-tokenizer-uds\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:11:38.303239 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.303071    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776c7f7b-85ca-438f-a932-3875fcf4e653-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:11:38.667862 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.667820    2582 generic.go:356] "Generic (PLEG): container finished" podID="776c7f7b-85ca-438f-a932-3875fcf4e653" containerID="198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155" exitCode=0
Jul 08 07:11:38.668028 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.667869    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"
Jul 08 07:11:38.668028 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.667882    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t" event={"ID":"776c7f7b-85ca-438f-a932-3875fcf4e653","Type":"ContainerDied","Data":"198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155"}
Jul 08 07:11:38.668028 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.667965    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t" event={"ID":"776c7f7b-85ca-438f-a932-3875fcf4e653","Type":"ContainerDied","Data":"e3d01b12de172af334df8cfe0c1ac630c935ca99242ffd49aad289483407f73f"}
Jul 08 07:11:38.668028 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.667989    2582 scope.go:117] "RemoveContainer" containerID="198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155"
Jul 08 07:11:38.677389 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.677373    2582 scope.go:117] "RemoveContainer" containerID="4598cb7fb05be881057c3f092cf2354faa5a2e823bc3f921607f39a45b82fe82"
Jul 08 07:11:38.684506 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.684488    2582 scope.go:117] "RemoveContainer" containerID="198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155"
Jul 08 07:11:38.684751 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:11:38.684734    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155\": container with ID starting with 198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155 not found: ID does not exist" containerID="198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155"
Jul 08 07:11:38.684797 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.684760    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155"} err="failed to get container status \"198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155\": rpc error: code = NotFound desc = could not find container \"198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155\": container with ID starting with 198884a47882a3a4bb65c9c910bf75ce793de16021703706ca2a0333d32c9155 not found: ID does not exist"
Jul 08 07:11:38.684797 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.684777    2582 scope.go:117] "RemoveContainer" containerID="4598cb7fb05be881057c3f092cf2354faa5a2e823bc3f921607f39a45b82fe82"
Jul 08 07:11:38.684973 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:11:38.684956    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4598cb7fb05be881057c3f092cf2354faa5a2e823bc3f921607f39a45b82fe82\": container with ID starting with 4598cb7fb05be881057c3f092cf2354faa5a2e823bc3f921607f39a45b82fe82 not found: ID does not exist" containerID="4598cb7fb05be881057c3f092cf2354faa5a2e823bc3f921607f39a45b82fe82"
Jul 08 07:11:38.685014 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.684980    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4598cb7fb05be881057c3f092cf2354faa5a2e823bc3f921607f39a45b82fe82"} err="failed to get container status \"4598cb7fb05be881057c3f092cf2354faa5a2e823bc3f921607f39a45b82fe82\": rpc error: code = NotFound desc = could not find container \"4598cb7fb05be881057c3f092cf2354faa5a2e823bc3f921607f39a45b82fe82\": container with ID starting with 4598cb7fb05be881057c3f092cf2354faa5a2e823bc3f921607f39a45b82fe82 not found: ID does not exist"
Jul 08 07:11:38.688911 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.688889    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"]
Jul 08 07:11:38.692324 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.692303    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-869bfb7bcf-46w7t"]
Jul 08 07:11:38.745259 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:38.745233    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776c7f7b-85ca-438f-a932-3875fcf4e653" path="/var/lib/kubelet/pods/776c7f7b-85ca-438f-a932-3875fcf4e653/volumes"
Jul 08 07:11:44.968538 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:44.968466    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:11:45.429560 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:45.429524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:11:52.877631 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:52.877597    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main" containerID="cri-o://f32eb32767656d445093f5981c4f9399dcdccc0a98a6acf8033845f41e54943c" gracePeriod=45
Jul 08 07:11:53.624263 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.624241    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:11:53.712885 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.712860    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-kserve-provision-location\") pod \"57f5b1d9-b254-4410-8937-06a6da16c06a\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") "
Jul 08 07:11:53.713014 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.712899    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-tmp-dir\") pod \"57f5b1d9-b254-4410-8937-06a6da16c06a\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") "
Jul 08 07:11:53.713014 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.712913    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-model-cache\") pod \"57f5b1d9-b254-4410-8937-06a6da16c06a\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") "
Jul 08 07:11:53.713014 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.712949    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-home\") pod \"57f5b1d9-b254-4410-8937-06a6da16c06a\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") "
Jul 08 07:11:53.713014 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.712975    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvhgz\" (UniqueName: \"kubernetes.io/projected/57f5b1d9-b254-4410-8937-06a6da16c06a-kube-api-access-gvhgz\") pod \"57f5b1d9-b254-4410-8937-06a6da16c06a\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") "
Jul 08 07:11:53.713014 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.712993    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-dshm\") pod \"57f5b1d9-b254-4410-8937-06a6da16c06a\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") "
Jul 08 07:11:53.713353 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.713036    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f5b1d9-b254-4410-8937-06a6da16c06a-tls-certs\") pod \"57f5b1d9-b254-4410-8937-06a6da16c06a\" (UID: \"57f5b1d9-b254-4410-8937-06a6da16c06a\") "
Jul 08 07:11:53.713353 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.713242    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-model-cache" (OuterVolumeSpecName: "model-cache") pod "57f5b1d9-b254-4410-8937-06a6da16c06a" (UID: "57f5b1d9-b254-4410-8937-06a6da16c06a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:11:53.713836 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.713812    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-home" (OuterVolumeSpecName: "home") pod "57f5b1d9-b254-4410-8937-06a6da16c06a" (UID: "57f5b1d9-b254-4410-8937-06a6da16c06a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:11:53.715228 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.715201    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f5b1d9-b254-4410-8937-06a6da16c06a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57f5b1d9-b254-4410-8937-06a6da16c06a" (UID: "57f5b1d9-b254-4410-8937-06a6da16c06a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:11:53.715639 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.715616    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f5b1d9-b254-4410-8937-06a6da16c06a-kube-api-access-gvhgz" (OuterVolumeSpecName: "kube-api-access-gvhgz") pod "57f5b1d9-b254-4410-8937-06a6da16c06a" (UID: "57f5b1d9-b254-4410-8937-06a6da16c06a"). InnerVolumeSpecName "kube-api-access-gvhgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:11:53.715720 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.715654    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-dshm" (OuterVolumeSpecName: "dshm") pod "57f5b1d9-b254-4410-8937-06a6da16c06a" (UID: "57f5b1d9-b254-4410-8937-06a6da16c06a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:11:53.725635 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.725611    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57f5b1d9-b254-4410-8937-06a6da16c06a" (UID: "57f5b1d9-b254-4410-8937-06a6da16c06a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:11:53.729688 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.729661    2582 generic.go:356] "Generic (PLEG): container finished" podID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerID="f32eb32767656d445093f5981c4f9399dcdccc0a98a6acf8033845f41e54943c" exitCode=0
Jul 08 07:11:53.729777 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.729708    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"
Jul 08 07:11:53.729777 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.729717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" event={"ID":"57f5b1d9-b254-4410-8937-06a6da16c06a","Type":"ContainerDied","Data":"f32eb32767656d445093f5981c4f9399dcdccc0a98a6acf8033845f41e54943c"}
Jul 08 07:11:53.729777 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.729766    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg" event={"ID":"57f5b1d9-b254-4410-8937-06a6da16c06a","Type":"ContainerDied","Data":"b64050df225ceff9b79da2a2514b09a7d3e4c9d10fa2cf657c767a66793fc4b5"}
Jul 08 07:11:53.729885 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.729789    2582 scope.go:117] "RemoveContainer" containerID="f32eb32767656d445093f5981c4f9399dcdccc0a98a6acf8033845f41e54943c"
Jul 08 07:11:53.737536 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.737515    2582 scope.go:117] "RemoveContainer" containerID="c47d6f51c9884954d93aad88bbe02695cba80233e6ed56c9ef83db18a19bee2e"
Jul 08 07:11:53.744998 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.744979    2582 scope.go:117] "RemoveContainer" containerID="f32eb32767656d445093f5981c4f9399dcdccc0a98a6acf8033845f41e54943c"
Jul 08 07:11:53.745264 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:11:53.745246    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32eb32767656d445093f5981c4f9399dcdccc0a98a6acf8033845f41e54943c\": container with ID starting with f32eb32767656d445093f5981c4f9399dcdccc0a98a6acf8033845f41e54943c not found: ID does not exist" containerID="f32eb32767656d445093f5981c4f9399dcdccc0a98a6acf8033845f41e54943c"
Jul 08 07:11:53.745320 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.745273    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32eb32767656d445093f5981c4f9399dcdccc0a98a6acf8033845f41e54943c"} err="failed to get container status \"f32eb32767656d445093f5981c4f9399dcdccc0a98a6acf8033845f41e54943c\": rpc error: code = NotFound desc = could not find container \"f32eb32767656d445093f5981c4f9399dcdccc0a98a6acf8033845f41e54943c\": container with ID starting with f32eb32767656d445093f5981c4f9399dcdccc0a98a6acf8033845f41e54943c not found: ID does not exist"
Jul 08 07:11:53.745320 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.745289    2582 scope.go:117] "RemoveContainer" containerID="c47d6f51c9884954d93aad88bbe02695cba80233e6ed56c9ef83db18a19bee2e"
Jul 08 07:11:53.745537 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:11:53.745515    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47d6f51c9884954d93aad88bbe02695cba80233e6ed56c9ef83db18a19bee2e\": container with ID starting with c47d6f51c9884954d93aad88bbe02695cba80233e6ed56c9ef83db18a19bee2e not found: ID does not exist" containerID="c47d6f51c9884954d93aad88bbe02695cba80233e6ed56c9ef83db18a19bee2e"
Jul 08 07:11:53.745577 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.745547    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47d6f51c9884954d93aad88bbe02695cba80233e6ed56c9ef83db18a19bee2e"} err="failed to get container status \"c47d6f51c9884954d93aad88bbe02695cba80233e6ed56c9ef83db18a19bee2e\": rpc error: code = NotFound desc = could not find container \"c47d6f51c9884954d93aad88bbe02695cba80233e6ed56c9ef83db18a19bee2e\": container with ID starting with c47d6f51c9884954d93aad88bbe02695cba80233e6ed56c9ef83db18a19bee2e not found: ID does not exist"
Jul 08 07:11:53.774932 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.774907    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57f5b1d9-b254-4410-8937-06a6da16c06a" (UID: "57f5b1d9-b254-4410-8937-06a6da16c06a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:11:53.814026 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.814004    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f5b1d9-b254-4410-8937-06a6da16c06a-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:11:53.814026 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.814024    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:11:53.814172 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.814035    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:11:53.814172 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.814045    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:11:53.814172 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.814053    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:11:53.814172 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.814061    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvhgz\" (UniqueName: \"kubernetes.io/projected/57f5b1d9-b254-4410-8937-06a6da16c06a-kube-api-access-gvhgz\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:11:53.814172 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:53.814068    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f5b1d9-b254-4410-8937-06a6da16c06a-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:11:54.051752 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:54.051730    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"]
Jul 08 07:11:54.055660 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:54.055638    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-b46fb9c55-wbzwg"]
Jul 08 07:11:54.749239 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:54.749206    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" path="/var/lib/kubelet/pods/57f5b1d9-b254-4410-8937-06a6da16c06a/volumes"
Jul 08 07:11:59.318014 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:11:59.317962    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:12:05.390511 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:05.390448    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:12:23.393914 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.393878    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"]
Jul 08 07:12:23.394519 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.394143    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776c7f7b-85ca-438f-a932-3875fcf4e653" containerName="main"
Jul 08 07:12:23.394519 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.394177    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="776c7f7b-85ca-438f-a932-3875fcf4e653" containerName="main"
Jul 08 07:12:23.394519 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.394187    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="storage-initializer"
Jul 08 07:12:23.394519 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.394192    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="storage-initializer"
Jul 08 07:12:23.394519 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.394205    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main"
Jul 08 07:12:23.394519 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.394211    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main"
Jul 08 07:12:23.394519 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.394217    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776c7f7b-85ca-438f-a932-3875fcf4e653" containerName="storage-initializer"
Jul 08 07:12:23.394519 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.394222    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="776c7f7b-85ca-438f-a932-3875fcf4e653" containerName="storage-initializer"
Jul 08 07:12:23.394519 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.394265    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="776c7f7b-85ca-438f-a932-3875fcf4e653" containerName="main"
Jul 08 07:12:23.394519 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.394271    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="57f5b1d9-b254-4410-8937-06a6da16c06a" containerName="main"
Jul 08 07:12:23.400975 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.400948    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.403526 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.403500    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 07:12:23.403703 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.403536    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 07:12:23.403703 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.403582    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 08 07:12:23.403868 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.403628    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-psjvw\""
Jul 08 07:12:23.403868 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.403635    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 07:12:23.409728 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.409704    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"]
Jul 08 07:12:23.431331 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.431308    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:12:23.521651 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.521626    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-home\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.521756 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.521654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-dshm\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.521756 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.521676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg8dq\" (UniqueName: \"kubernetes.io/projected/1c9529db-28e9-461a-b56d-d893239988f0-kube-api-access-lg8dq\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.521756 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.521703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.521756 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.521731    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9529db-28e9-461a-b56d-d893239988f0-tls-certs\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.521888 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.521780    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-model-cache\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.521888 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.521821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-tmp-dir\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.622140 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.622115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.622261 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.622142    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9529db-28e9-461a-b56d-d893239988f0-tls-certs\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.622261 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.622204    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-model-cache\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.622261 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.622226    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-tmp-dir\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.622261 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.622255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-home\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.622455 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.622275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-dshm\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.622455 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.622298    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg8dq\" (UniqueName: \"kubernetes.io/projected/1c9529db-28e9-461a-b56d-d893239988f0-kube-api-access-lg8dq\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.622601 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.622579    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-tmp-dir\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.622717 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.622591    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-model-cache\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.622717 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.622657    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-home\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.622790 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.622752    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.624456 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.624440    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-dshm\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.624640 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.624624    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9529db-28e9-461a-b56d-d893239988f0-tls-certs\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.630366 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.630346    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg8dq\" (UniqueName: \"kubernetes.io/projected/1c9529db-28e9-461a-b56d-d893239988f0-kube-api-access-lg8dq\") pod \"auth-invalid-token-test-kserve-78f6c8fccf-rtmzj\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.703216 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.703192    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"]
Jul 08 07:12:23.708067 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.708049    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:23.710552 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.710534    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-njxgn\""
Jul 08 07:12:23.712429 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.712301    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:23.719511 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.719490    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"]
Jul 08 07:12:23.743846 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.743825    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:12:23.823706 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.823670    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f29b06f2-41a0-44d0-b7d6-3f8e22804271-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:23.823842 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.823773    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29b06f2-41a0-44d0-b7d6-3f8e22804271-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:23.823904 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.823875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvxmt\" (UniqueName: \"kubernetes.io/projected/f29b06f2-41a0-44d0-b7d6-3f8e22804271-kube-api-access-fvxmt\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:23.823967 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.823918    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f29b06f2-41a0-44d0-b7d6-3f8e22804271-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:23.845186 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.845145    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"]
Jul 08 07:12:23.847922 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:12:23.847896    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9529db_28e9_461a_b56d_d893239988f0.slice/crio-8610e09ae45b202c5e07133b06815423e2dbe30b81612f20f15220c765711944 WatchSource:0}: Error finding container 8610e09ae45b202c5e07133b06815423e2dbe30b81612f20f15220c765711944: Status 404 returned error can't find the container with id 8610e09ae45b202c5e07133b06815423e2dbe30b81612f20f15220c765711944
Jul 08 07:12:23.924878 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.924854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvxmt\" (UniqueName: \"kubernetes.io/projected/f29b06f2-41a0-44d0-b7d6-3f8e22804271-kube-api-access-fvxmt\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:23.924988 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.924897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f29b06f2-41a0-44d0-b7d6-3f8e22804271-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:23.924988 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.924941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f29b06f2-41a0-44d0-b7d6-3f8e22804271-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:23.925102 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.924992    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29b06f2-41a0-44d0-b7d6-3f8e22804271-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:23.925294 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.925275    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f29b06f2-41a0-44d0-b7d6-3f8e22804271-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:23.925372 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.925299    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f29b06f2-41a0-44d0-b7d6-3f8e22804271-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:23.927068 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.927048    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29b06f2-41a0-44d0-b7d6-3f8e22804271-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:23.934774 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:23.934750    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvxmt\" (UniqueName: \"kubernetes.io/projected/f29b06f2-41a0-44d0-b7d6-3f8e22804271-kube-api-access-fvxmt\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:24.019953 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:24.019881    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:24.181083 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:24.181039    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"]
Jul 08 07:12:24.182809 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:12:24.182764    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29b06f2_41a0_44d0_b7d6_3f8e22804271.slice/crio-f53bd6feef82473b0dd06e6eb085e0b2bf1f077b519e9476f338b637c75a4197 WatchSource:0}: Error finding container f53bd6feef82473b0dd06e6eb085e0b2bf1f077b519e9476f338b637c75a4197: Status 404 returned error can't find the container with id f53bd6feef82473b0dd06e6eb085e0b2bf1f077b519e9476f338b637c75a4197
Jul 08 07:12:24.852141 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:24.852108    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht" event={"ID":"f29b06f2-41a0-44d0-b7d6-3f8e22804271","Type":"ContainerStarted","Data":"135e2cfed2289fe6f2191725dd2b867e80745916b60582f5f18002f168c8e757"}
Jul 08 07:12:24.852618 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:24.852175    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht" event={"ID":"f29b06f2-41a0-44d0-b7d6-3f8e22804271","Type":"ContainerStarted","Data":"f53bd6feef82473b0dd06e6eb085e0b2bf1f077b519e9476f338b637c75a4197"}
Jul 08 07:12:24.853847 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:24.853820    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" event={"ID":"1c9529db-28e9-461a-b56d-d893239988f0","Type":"ContainerStarted","Data":"713fb30f9f09d504f878263b2e1a021a6bc0f3a185bf7b1f10d0c1b0637c3e2d"}
Jul 08 07:12:24.853974 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:24.853870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" event={"ID":"1c9529db-28e9-461a-b56d-d893239988f0","Type":"ContainerStarted","Data":"8610e09ae45b202c5e07133b06815423e2dbe30b81612f20f15220c765711944"}
Jul 08 07:12:25.860079 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:25.860040    2582 generic.go:356] "Generic (PLEG): container finished" podID="f29b06f2-41a0-44d0-b7d6-3f8e22804271" containerID="135e2cfed2289fe6f2191725dd2b867e80745916b60582f5f18002f168c8e757" exitCode=0
Jul 08 07:12:25.860548 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:25.860134    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht" event={"ID":"f29b06f2-41a0-44d0-b7d6-3f8e22804271","Type":"ContainerDied","Data":"135e2cfed2289fe6f2191725dd2b867e80745916b60582f5f18002f168c8e757"}
Jul 08 07:12:26.870653 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:26.870573    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht" event={"ID":"f29b06f2-41a0-44d0-b7d6-3f8e22804271","Type":"ContainerStarted","Data":"0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36"}
Jul 08 07:12:26.871133 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:26.870804    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:12:26.895099 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:26.895047    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht" podStartSLOduration=3.099079708 podStartE2EDuration="3.895032218s" totalImagesPullingTime="0s" totalInitContainerRuntime="795.95251ms" isStatefulPod=false podCreationTimestamp="2026-07-08 07:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:12:26.89347511 +0000 UTC m=+1062.697261511" watchObservedRunningTime="2026-07-08 07:12:26.895032218 +0000 UTC m=+1062.698818617"
Jul 08 07:12:27.876523 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:27.876493    2582 generic.go:356] "Generic (PLEG): container finished" podID="1c9529db-28e9-461a-b56d-d893239988f0" containerID="713fb30f9f09d504f878263b2e1a021a6bc0f3a185bf7b1f10d0c1b0637c3e2d" exitCode=0
Jul 08 07:12:27.876899 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:27.876569    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" event={"ID":"1c9529db-28e9-461a-b56d-d893239988f0","Type":"ContainerDied","Data":"713fb30f9f09d504f878263b2e1a021a6bc0f3a185bf7b1f10d0c1b0637c3e2d"}
Jul 08 07:12:28.884386 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:28.884348    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" event={"ID":"1c9529db-28e9-461a-b56d-d893239988f0","Type":"ContainerStarted","Data":"7eb5cd77efe6a051aeb654e7a9d403f5b06cf448d92c8dbdac0f7fc7222bbc0d"}
Jul 08 07:12:28.910668 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:28.910589    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" podStartSLOduration=2.466637441 podStartE2EDuration="5.910566345s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.443928904s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:12:28.907845967 +0000 UTC m=+1064.711632398" watchObservedRunningTime="2026-07-08 07:12:28.910566345 +0000 UTC m=+1064.714352745"
Jul 08 07:12:29.712516 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:29.712480    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:32.385868 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:32.385832    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:12:33.713043 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:33.713006    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:12:33.714529 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:33.714497    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 08 07:12:34.493199 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:34.493128    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:12:34.556513 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:34.556475    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:12:37.887720 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:37.887638    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:12:43.713701 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:43.713657    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 08 07:12:53.662749 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:53.662700    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:12:53.713590 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:53.713538    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 08 07:12:53.766680 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:53.766647    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:12:57.878940 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:12:57.878909    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:13:03.713494 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:13:03.713453    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 08 07:13:13.713229 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:13:13.713188    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 08 07:13:23.713535 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:13:23.713495    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 08 07:13:24.015586 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:13:24.015511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:13:24.387423 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:13:24.387339    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:13:33.713551 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:13:33.713509    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 08 07:13:43.712825 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:13:43.712776    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 08 07:13:43.822535 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:13:43.822503    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:13:53.712922 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:13:53.712877    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 08 07:14:03.713108 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:03.713055    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 08 07:14:13.712772 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:13.712683    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 08 07:14:23.722914 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:23.722883    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:14:23.730426 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:23.730407    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:14:25.874043 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:25.874013    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:14:25.874726 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:25.874705    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"]
Jul 08 07:14:25.888172 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:25.888129    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"]
Jul 08 07:14:25.896514 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:14:25.896483    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 07:14:25.896514 ip-10-0-140-110 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 07:14:25.896514 ip-10-0-140-110 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 08 07:14:25.896514 ip-10-0-140-110 kubenswrapper[2582]:  > containerID="0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36" cmd=["/bin/sleep","15"]
Jul 08 07:14:25.896776 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:25.896538    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht" message="" err=<
Jul 08 07:14:25.896776 ip-10-0-140-110 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 07:14:25.896776 ip-10-0-140-110 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 08 07:14:25.896776 ip-10-0-140-110 kubenswrapper[2582]:  >
Jul 08 07:14:25.896776 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:14:25.896554    2582 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 07:14:25.896776 ip-10-0-140-110 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 07:14:25.896776 ip-10-0-140-110 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Jul 08 07:14:25.896776 ip-10-0-140-110 kubenswrapper[2582]:  > pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht" podUID="f29b06f2-41a0-44d0-b7d6-3f8e22804271" containerName="main" containerID="cri-o://0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36"
Jul 08 07:14:25.896776 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:25.896597    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht" podUID="f29b06f2-41a0-44d0-b7d6-3f8e22804271" containerName="main" containerID="cri-o://0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36" gracePeriod=60
Jul 08 07:14:26.048644 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.048616    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:14:26.139387 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.139363    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:14:26.321890 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.321862    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f29b06f2-41a0-44d0-b7d6-3f8e22804271-tokenizer-uds\") pod \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") "
Jul 08 07:14:26.322051 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.321903    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29b06f2-41a0-44d0-b7d6-3f8e22804271-tls-certs\") pod \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") "
Jul 08 07:14:26.322051 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.321934    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f29b06f2-41a0-44d0-b7d6-3f8e22804271-kserve-provision-location\") pod \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") "
Jul 08 07:14:26.322051 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.321956    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvxmt\" (UniqueName: \"kubernetes.io/projected/f29b06f2-41a0-44d0-b7d6-3f8e22804271-kube-api-access-fvxmt\") pod \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\" (UID: \"f29b06f2-41a0-44d0-b7d6-3f8e22804271\") "
Jul 08 07:14:26.322241 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.322199    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29b06f2-41a0-44d0-b7d6-3f8e22804271-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f29b06f2-41a0-44d0-b7d6-3f8e22804271" (UID: "f29b06f2-41a0-44d0-b7d6-3f8e22804271"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:14:26.322576 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.322556    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29b06f2-41a0-44d0-b7d6-3f8e22804271-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f29b06f2-41a0-44d0-b7d6-3f8e22804271" (UID: "f29b06f2-41a0-44d0-b7d6-3f8e22804271"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:14:26.323958 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.323942    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29b06f2-41a0-44d0-b7d6-3f8e22804271-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f29b06f2-41a0-44d0-b7d6-3f8e22804271" (UID: "f29b06f2-41a0-44d0-b7d6-3f8e22804271"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:14:26.324041 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.324027    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29b06f2-41a0-44d0-b7d6-3f8e22804271-kube-api-access-fvxmt" (OuterVolumeSpecName: "kube-api-access-fvxmt") pod "f29b06f2-41a0-44d0-b7d6-3f8e22804271" (UID: "f29b06f2-41a0-44d0-b7d6-3f8e22804271"). InnerVolumeSpecName "kube-api-access-fvxmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:14:26.408174 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.408091    2582 generic.go:356] "Generic (PLEG): container finished" podID="f29b06f2-41a0-44d0-b7d6-3f8e22804271" containerID="0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36" exitCode=0
Jul 08 07:14:26.408174 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.408144    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"
Jul 08 07:14:26.408315 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.408183    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht" event={"ID":"f29b06f2-41a0-44d0-b7d6-3f8e22804271","Type":"ContainerDied","Data":"0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36"}
Jul 08 07:14:26.408315 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.408232    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht" event={"ID":"f29b06f2-41a0-44d0-b7d6-3f8e22804271","Type":"ContainerDied","Data":"f53bd6feef82473b0dd06e6eb085e0b2bf1f077b519e9476f338b637c75a4197"}
Jul 08 07:14:26.408315 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.408254    2582 scope.go:117] "RemoveContainer" containerID="0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36"
Jul 08 07:14:26.417774 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.417758    2582 scope.go:117] "RemoveContainer" containerID="135e2cfed2289fe6f2191725dd2b867e80745916b60582f5f18002f168c8e757"
Jul 08 07:14:26.423140 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.423120    2582 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f29b06f2-41a0-44d0-b7d6-3f8e22804271-tokenizer-uds\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:14:26.423140 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.423138    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29b06f2-41a0-44d0-b7d6-3f8e22804271-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:14:26.423140 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.423146    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f29b06f2-41a0-44d0-b7d6-3f8e22804271-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:14:26.423361 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.423169    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvxmt\" (UniqueName: \"kubernetes.io/projected/f29b06f2-41a0-44d0-b7d6-3f8e22804271-kube-api-access-fvxmt\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:14:26.424997 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.424979    2582 scope.go:117] "RemoveContainer" containerID="0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36"
Jul 08 07:14:26.425276 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:14:26.425257    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36\": container with ID starting with 0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36 not found: ID does not exist" containerID="0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36"
Jul 08 07:14:26.425328 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.425284    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36"} err="failed to get container status \"0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36\": rpc error: code = NotFound desc = could not find container \"0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36\": container with ID starting with 0203af08407dae84475d5669416f3d5f9c37a814d6dbe5a13ee0d34b99e14c36 not found: ID does not exist"
Jul 08 07:14:26.425328 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.425298    2582 scope.go:117] "RemoveContainer" containerID="135e2cfed2289fe6f2191725dd2b867e80745916b60582f5f18002f168c8e757"
Jul 08 07:14:26.425542 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:14:26.425525    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135e2cfed2289fe6f2191725dd2b867e80745916b60582f5f18002f168c8e757\": container with ID starting with 135e2cfed2289fe6f2191725dd2b867e80745916b60582f5f18002f168c8e757 not found: ID does not exist" containerID="135e2cfed2289fe6f2191725dd2b867e80745916b60582f5f18002f168c8e757"
Jul 08 07:14:26.425588 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.425548    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135e2cfed2289fe6f2191725dd2b867e80745916b60582f5f18002f168c8e757"} err="failed to get container status \"135e2cfed2289fe6f2191725dd2b867e80745916b60582f5f18002f168c8e757\": rpc error: code = NotFound desc = could not find container \"135e2cfed2289fe6f2191725dd2b867e80745916b60582f5f18002f168c8e757\": container with ID starting with 135e2cfed2289fe6f2191725dd2b867e80745916b60582f5f18002f168c8e757 not found: ID does not exist"
Jul 08 07:14:26.455353 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.455330    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"]
Jul 08 07:14:26.462675 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.462656    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b47569d45lc9ht"]
Jul 08 07:14:26.746117 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:26.746081    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f29b06f2-41a0-44d0-b7d6-3f8e22804271" path="/var/lib/kubelet/pods/f29b06f2-41a0-44d0-b7d6-3f8e22804271/volumes"
Jul 08 07:14:40.898659 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:40.898595    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main" containerID="cri-o://7eb5cd77efe6a051aeb654e7a9d403f5b06cf448d92c8dbdac0f7fc7222bbc0d" gracePeriod=45
Jul 08 07:14:41.477778 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.477743    2582 generic.go:356] "Generic (PLEG): container finished" podID="1c9529db-28e9-461a-b56d-d893239988f0" containerID="7eb5cd77efe6a051aeb654e7a9d403f5b06cf448d92c8dbdac0f7fc7222bbc0d" exitCode=0
Jul 08 07:14:41.477925 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.477810    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" event={"ID":"1c9529db-28e9-461a-b56d-d893239988f0","Type":"ContainerDied","Data":"7eb5cd77efe6a051aeb654e7a9d403f5b06cf448d92c8dbdac0f7fc7222bbc0d"}
Jul 08 07:14:41.552217 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.552187    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:14:41.627804 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.627773    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-tmp-dir\") pod \"1c9529db-28e9-461a-b56d-d893239988f0\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") "
Jul 08 07:14:41.627804 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.627810    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-dshm\") pod \"1c9529db-28e9-461a-b56d-d893239988f0\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") "
Jul 08 07:14:41.628055 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.627832    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9529db-28e9-461a-b56d-d893239988f0-tls-certs\") pod \"1c9529db-28e9-461a-b56d-d893239988f0\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") "
Jul 08 07:14:41.628055 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.627852    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-kserve-provision-location\") pod \"1c9529db-28e9-461a-b56d-d893239988f0\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") "
Jul 08 07:14:41.628055 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.627883    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg8dq\" (UniqueName: \"kubernetes.io/projected/1c9529db-28e9-461a-b56d-d893239988f0-kube-api-access-lg8dq\") pod \"1c9529db-28e9-461a-b56d-d893239988f0\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") "
Jul 08 07:14:41.628055 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.627901    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-model-cache\") pod \"1c9529db-28e9-461a-b56d-d893239988f0\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") "
Jul 08 07:14:41.628055 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.627951    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-home\") pod \"1c9529db-28e9-461a-b56d-d893239988f0\" (UID: \"1c9529db-28e9-461a-b56d-d893239988f0\") "
Jul 08 07:14:41.628392 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.628223    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c9529db-28e9-461a-b56d-d893239988f0" (UID: "1c9529db-28e9-461a-b56d-d893239988f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:14:41.628757 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.628731    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-home" (OuterVolumeSpecName: "home") pod "1c9529db-28e9-461a-b56d-d893239988f0" (UID: "1c9529db-28e9-461a-b56d-d893239988f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:14:41.630003 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.629980    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-dshm" (OuterVolumeSpecName: "dshm") pod "1c9529db-28e9-461a-b56d-d893239988f0" (UID: "1c9529db-28e9-461a-b56d-d893239988f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:14:41.630370 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.630342    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9529db-28e9-461a-b56d-d893239988f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c9529db-28e9-461a-b56d-d893239988f0" (UID: "1c9529db-28e9-461a-b56d-d893239988f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:14:41.630465 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.630370    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9529db-28e9-461a-b56d-d893239988f0-kube-api-access-lg8dq" (OuterVolumeSpecName: "kube-api-access-lg8dq") pod "1c9529db-28e9-461a-b56d-d893239988f0" (UID: "1c9529db-28e9-461a-b56d-d893239988f0"). InnerVolumeSpecName "kube-api-access-lg8dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:14:41.639725 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.639698    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c9529db-28e9-461a-b56d-d893239988f0" (UID: "1c9529db-28e9-461a-b56d-d893239988f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:14:41.688099 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.688056    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c9529db-28e9-461a-b56d-d893239988f0" (UID: "1c9529db-28e9-461a-b56d-d893239988f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:14:41.728792 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.728762    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:14:41.728792 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.728788    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:14:41.728921 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.728798    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9529db-28e9-461a-b56d-d893239988f0-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:14:41.728921 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.728807    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:14:41.728921 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.728817    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg8dq\" (UniqueName: \"kubernetes.io/projected/1c9529db-28e9-461a-b56d-d893239988f0-kube-api-access-lg8dq\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:14:41.728921 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.728825    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:14:41.728921 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:41.728833    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c9529db-28e9-461a-b56d-d893239988f0-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:14:42.483282 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:42.483237    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj" event={"ID":"1c9529db-28e9-461a-b56d-d893239988f0","Type":"ContainerDied","Data":"8610e09ae45b202c5e07133b06815423e2dbe30b81612f20f15220c765711944"}
Jul 08 07:14:42.483282 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:42.483276    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"
Jul 08 07:14:42.483777 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:42.483312    2582 scope.go:117] "RemoveContainer" containerID="7eb5cd77efe6a051aeb654e7a9d403f5b06cf448d92c8dbdac0f7fc7222bbc0d"
Jul 08 07:14:42.493992 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:42.493972    2582 scope.go:117] "RemoveContainer" containerID="713fb30f9f09d504f878263b2e1a021a6bc0f3a185bf7b1f10d0c1b0637c3e2d"
Jul 08 07:14:42.511402 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:42.511381    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"]
Jul 08 07:14:42.517575 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:42.517555    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-78f6c8fccf-rtmzj"]
Jul 08 07:14:42.745392 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:42.745326    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c9529db-28e9-461a-b56d-d893239988f0" path="/var/lib/kubelet/pods/1c9529db-28e9-461a-b56d-d893239988f0/volumes"
Jul 08 07:14:43.721959 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:43.721910    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:14:44.685808 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:44.685776    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:14:44.688222 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:44.688200    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:14:46.432440 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.432406    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"]
Jul 08 07:14:46.432805 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.432669    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29b06f2-41a0-44d0-b7d6-3f8e22804271" containerName="storage-initializer"
Jul 08 07:14:46.432805 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.432681    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29b06f2-41a0-44d0-b7d6-3f8e22804271" containerName="storage-initializer"
Jul 08 07:14:46.432805 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.432690    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main"
Jul 08 07:14:46.432805 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.432695    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main"
Jul 08 07:14:46.432805 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.432708    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29b06f2-41a0-44d0-b7d6-3f8e22804271" containerName="main"
Jul 08 07:14:46.432805 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.432714    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29b06f2-41a0-44d0-b7d6-3f8e22804271" containerName="main"
Jul 08 07:14:46.432805 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.432721    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="storage-initializer"
Jul 08 07:14:46.432805 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.432726    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="storage-initializer"
Jul 08 07:14:46.432805 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.432788    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c9529db-28e9-461a-b56d-d893239988f0" containerName="main"
Jul 08 07:14:46.432805 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.432795    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f29b06f2-41a0-44d0-b7d6-3f8e22804271" containerName="main"
Jul 08 07:14:46.436795 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.436779    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.440603 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.440581    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 07:14:46.441828 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.441799    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 07:14:46.441993 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.441865    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 08 07:14:46.441993 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.441900    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-psjvw\""
Jul 08 07:14:46.455641 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.455623    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"]
Jul 08 07:14:46.456017 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.455994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.456109 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.456021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.456109 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.456037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776effdc-0b77-4070-8301-97e01a029d49-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.456109 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.456057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vm8l\" (UniqueName: \"kubernetes.io/projected/776effdc-0b77-4070-8301-97e01a029d49-kube-api-access-8vm8l\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.456234 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.456126    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.456234 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.456183    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.503339 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.503309    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:14:46.556953 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.556924    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.556953 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.556953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.557106 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.556973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776effdc-0b77-4070-8301-97e01a029d49-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.557229 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.557190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vm8l\" (UniqueName: \"kubernetes.io/projected/776effdc-0b77-4070-8301-97e01a029d49-kube-api-access-8vm8l\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.557325 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.557303    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.557384 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.557363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.557436 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.557397    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.557645 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.557626    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.557701 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.557664    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.559051 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.559032    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.559343 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.559325    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776effdc-0b77-4070-8301-97e01a029d49-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.569465 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.569439    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vm8l\" (UniqueName: \"kubernetes.io/projected/776effdc-0b77-4070-8301-97e01a029d49-kube-api-access-8vm8l\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.747550 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.747466    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:46.885344 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:46.885319    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"]
Jul 08 07:14:46.887238 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:14:46.887213    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776effdc_0b77_4070_8301_97e01a029d49.slice/crio-0552e02dd5d815ff6dc148b4ea5389f302a96b6d7a37d4f261e8df650d549ea1 WatchSource:0}: Error finding container 0552e02dd5d815ff6dc148b4ea5389f302a96b6d7a37d4f261e8df650d549ea1: Status 404 returned error can't find the container with id 0552e02dd5d815ff6dc148b4ea5389f302a96b6d7a37d4f261e8df650d549ea1
Jul 08 07:14:47.103038 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:47.102963    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:14:47.506176 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:47.506133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx" event={"ID":"776effdc-0b77-4070-8301-97e01a029d49","Type":"ContainerStarted","Data":"be68e25a2ebfc593faeba4a68857260ffb95d4f752dcf80a8b226cdda5af2d2a"}
Jul 08 07:14:47.506540 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:47.506201    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx" event={"ID":"776effdc-0b77-4070-8301-97e01a029d49","Type":"ContainerStarted","Data":"0552e02dd5d815ff6dc148b4ea5389f302a96b6d7a37d4f261e8df650d549ea1"}
Jul 08 07:14:47.560716 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:47.560669    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx" podStartSLOduration=1.560655676 podStartE2EDuration="1.560655676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:14:47.560056639 +0000 UTC m=+1203.363843059" watchObservedRunningTime="2026-07-08 07:14:47.560655676 +0000 UTC m=+1203.364442076"
Jul 08 07:14:47.747921 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:47.747890    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:55.304127 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.304095    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"]
Jul 08 07:14:55.307528 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.307510    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.311480 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.311454    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 08 07:14:55.311480 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.311476    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 07:14:55.322298 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.322276    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"]
Jul 08 07:14:55.324639 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.324618    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-model-cache\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.324736 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.324663    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-home\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.324736 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.324710    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s824\" (UniqueName: \"kubernetes.io/projected/275c2425-5390-4987-804a-f70b52e9dd60-kube-api-access-8s824\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.324849 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.324775    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275c2425-5390-4987-804a-f70b52e9dd60-tls-certs\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.324849 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.324824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-kserve-provision-location\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.324946 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.324854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-dshm\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.324946 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.324900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-tmp-dir\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.356937 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.356913    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:14:55.426077 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.426056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-model-cache\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.426190 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.426094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-home\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.426190 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.426116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s824\" (UniqueName: \"kubernetes.io/projected/275c2425-5390-4987-804a-f70b52e9dd60-kube-api-access-8s824\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.426348 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.426328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275c2425-5390-4987-804a-f70b52e9dd60-tls-certs\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.426409 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.426393    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-kserve-provision-location\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.426465 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.426426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-dshm\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.426517 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.426462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-tmp-dir\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.426517 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.426482    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-home\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.426625 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.426532    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-model-cache\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.426719 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.426697    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-kserve-provision-location\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.426809 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.426791    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-tmp-dir\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.428567 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.428546    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/275c2425-5390-4987-804a-f70b52e9dd60-dshm\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.428767 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.428749    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/275c2425-5390-4987-804a-f70b52e9dd60-tls-certs\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.435946 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.435926    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s824\" (UniqueName: \"kubernetes.io/projected/275c2425-5390-4987-804a-f70b52e9dd60-kube-api-access-8s824\") pod \"auth-disabled-test-kserve-68f5d8959c-xh45g\" (UID: \"275c2425-5390-4987-804a-f70b52e9dd60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.620417 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.620347    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:14:55.648837 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.648809    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"]
Jul 08 07:14:55.652131 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.652115    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:55.657474 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.657452    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-epp-sa-dockercfg-r47px\""
Jul 08 07:14:55.699572 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.699544    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"]
Jul 08 07:14:55.708229 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.708203    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:14:55.727955 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.727924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0f58936-1d64-48ee-90e4-40b8b1145eeb-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g\" (UID: \"b0f58936-1d64-48ee-90e4-40b8b1145eeb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:55.728085 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.727966    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f58936-1d64-48ee-90e4-40b8b1145eeb-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g\" (UID: \"b0f58936-1d64-48ee-90e4-40b8b1145eeb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:55.728085 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.728042    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbcrz\" (UniqueName: \"kubernetes.io/projected/b0f58936-1d64-48ee-90e4-40b8b1145eeb-kube-api-access-sbcrz\") pod \"auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g\" (UID: \"b0f58936-1d64-48ee-90e4-40b8b1145eeb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:55.728085 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.728078    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0f58936-1d64-48ee-90e4-40b8b1145eeb-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g\" (UID: \"b0f58936-1d64-48ee-90e4-40b8b1145eeb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:55.761584 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.761562    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"]
Jul 08 07:14:55.766541 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:14:55.766509    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275c2425_5390_4987_804a_f70b52e9dd60.slice/crio-e590c87afae1814b9bd3aee1dc9a3a367e87245db85e568f32d8a52284f72e68 WatchSource:0}: Error finding container e590c87afae1814b9bd3aee1dc9a3a367e87245db85e568f32d8a52284f72e68: Status 404 returned error can't find the container with id e590c87afae1814b9bd3aee1dc9a3a367e87245db85e568f32d8a52284f72e68
Jul 08 07:14:55.828852 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.828832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f58936-1d64-48ee-90e4-40b8b1145eeb-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g\" (UID: \"b0f58936-1d64-48ee-90e4-40b8b1145eeb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:55.828954 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.828886    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbcrz\" (UniqueName: \"kubernetes.io/projected/b0f58936-1d64-48ee-90e4-40b8b1145eeb-kube-api-access-sbcrz\") pod \"auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g\" (UID: \"b0f58936-1d64-48ee-90e4-40b8b1145eeb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:55.828954 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.828933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0f58936-1d64-48ee-90e4-40b8b1145eeb-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g\" (UID: \"b0f58936-1d64-48ee-90e4-40b8b1145eeb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:55.829053 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.829011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0f58936-1d64-48ee-90e4-40b8b1145eeb-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g\" (UID: \"b0f58936-1d64-48ee-90e4-40b8b1145eeb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:55.829330 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.829307    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0f58936-1d64-48ee-90e4-40b8b1145eeb-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g\" (UID: \"b0f58936-1d64-48ee-90e4-40b8b1145eeb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:55.829411 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.829368    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b0f58936-1d64-48ee-90e4-40b8b1145eeb-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g\" (UID: \"b0f58936-1d64-48ee-90e4-40b8b1145eeb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:55.831106 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.831087    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f58936-1d64-48ee-90e4-40b8b1145eeb-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g\" (UID: \"b0f58936-1d64-48ee-90e4-40b8b1145eeb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:55.851410 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.851388    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbcrz\" (UniqueName: \"kubernetes.io/projected/b0f58936-1d64-48ee-90e4-40b8b1145eeb-kube-api-access-sbcrz\") pod \"auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g\" (UID: \"b0f58936-1d64-48ee-90e4-40b8b1145eeb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:55.962043 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:55.962016    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:56.101309 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:56.101280    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"]
Jul 08 07:14:56.104569 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:14:56.104538    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f58936_1d64_48ee_90e4_40b8b1145eeb.slice/crio-309a0e3af528d8f4d65946b0598737aa6fc42090a5165c667269aa7cb0d75ca8 WatchSource:0}: Error finding container 309a0e3af528d8f4d65946b0598737aa6fc42090a5165c667269aa7cb0d75ca8: Status 404 returned error can't find the container with id 309a0e3af528d8f4d65946b0598737aa6fc42090a5165c667269aa7cb0d75ca8
Jul 08 07:14:56.545690 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:56.545633    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" event={"ID":"275c2425-5390-4987-804a-f70b52e9dd60","Type":"ContainerStarted","Data":"23bb9ea2aa028659882ccccb4e7044bd962e470f66de45ef1b29479af110f60b"}
Jul 08 07:14:56.546279 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:56.546252    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" event={"ID":"275c2425-5390-4987-804a-f70b52e9dd60","Type":"ContainerStarted","Data":"e590c87afae1814b9bd3aee1dc9a3a367e87245db85e568f32d8a52284f72e68"}
Jul 08 07:14:56.547446 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:56.547414    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g" event={"ID":"b0f58936-1d64-48ee-90e4-40b8b1145eeb","Type":"ContainerStarted","Data":"67fe7b435d75167fb6cad8724cee0589fa079c490b76eb61c465d3f48d2b5a33"}
Jul 08 07:14:56.547556 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:56.547469    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g" event={"ID":"b0f58936-1d64-48ee-90e4-40b8b1145eeb","Type":"ContainerStarted","Data":"309a0e3af528d8f4d65946b0598737aa6fc42090a5165c667269aa7cb0d75ca8"}
Jul 08 07:14:56.753645 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:56.753611    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:56.779803 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:56.779744    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:57.553113 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:57.553080    2582 generic.go:356] "Generic (PLEG): container finished" podID="b0f58936-1d64-48ee-90e4-40b8b1145eeb" containerID="67fe7b435d75167fb6cad8724cee0589fa079c490b76eb61c465d3f48d2b5a33" exitCode=0
Jul 08 07:14:57.553531 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:57.553183    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g" event={"ID":"b0f58936-1d64-48ee-90e4-40b8b1145eeb","Type":"ContainerDied","Data":"67fe7b435d75167fb6cad8724cee0589fa079c490b76eb61c465d3f48d2b5a33"}
Jul 08 07:14:57.565324 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:57.565300    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:14:58.559929 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:58.559893    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g" event={"ID":"b0f58936-1d64-48ee-90e4-40b8b1145eeb","Type":"ContainerStarted","Data":"f71b34a77ed9344f9c86cc6ba7ca22ec933a55f6a3ab320ce1b79866a42674d3"}
Jul 08 07:14:58.560552 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:58.560296    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:14:58.584321 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:14:58.584267    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g" podStartSLOduration=2.820093575 podStartE2EDuration="3.584248679s" totalImagesPullingTime="0s" totalInitContainerRuntime="764.155104ms" isStatefulPod=false podCreationTimestamp="2026-07-08 07:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:14:58.582876405 +0000 UTC m=+1214.386662806" watchObservedRunningTime="2026-07-08 07:14:58.584248679 +0000 UTC m=+1214.388035079"
Jul 08 07:15:01.575479 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:01.575446    2582 generic.go:356] "Generic (PLEG): container finished" podID="275c2425-5390-4987-804a-f70b52e9dd60" containerID="23bb9ea2aa028659882ccccb4e7044bd962e470f66de45ef1b29479af110f60b" exitCode=0
Jul 08 07:15:01.575982 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:01.575487    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" event={"ID":"275c2425-5390-4987-804a-f70b52e9dd60","Type":"ContainerDied","Data":"23bb9ea2aa028659882ccccb4e7044bd962e470f66de45ef1b29479af110f60b"}
Jul 08 07:15:02.584128 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:02.584097    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" event={"ID":"275c2425-5390-4987-804a-f70b52e9dd60","Type":"ContainerStarted","Data":"64b85f4df3d23c2ee6d0f99adcdcd2adac556b592bf194f03a5310372146755c"}
Jul 08 07:15:02.609055 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:02.609000    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" podStartSLOduration=2.720615639 podStartE2EDuration="7.608984471s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.888368832s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:15:02.606664972 +0000 UTC m=+1218.410451373" watchObservedRunningTime="2026-07-08 07:15:02.608984471 +0000 UTC m=+1218.412770869"
Jul 08 07:15:02.621358 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:02.621334    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:15:05.620786 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:05.620750    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:15:05.622373 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:05.622341    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" podUID="275c2425-5390-4987-804a-f70b52e9dd60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 08 07:15:11.303931 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:11.303893    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:15:15.620978 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:15.620933    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" podUID="275c2425-5390-4987-804a-f70b52e9dd60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 08 07:15:25.621216 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:25.621169    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" podUID="275c2425-5390-4987-804a-f70b52e9dd60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 08 07:15:30.571591 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:30.571561    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g"
Jul 08 07:15:35.620812 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:35.620773    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" podUID="275c2425-5390-4987-804a-f70b52e9dd60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 08 07:15:36.798714 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:36.798680    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:15:40.180109 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:40.180022    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"]
Jul 08 07:15:40.182429 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:40.182393    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:15:40.242701 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:40.242661    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:15:45.621253 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:45.621213    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" podUID="275c2425-5390-4987-804a-f70b52e9dd60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 08 07:15:55.204167 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.204082    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx" podUID="776effdc-0b77-4070-8301-97e01a029d49" containerName="main" containerID="cri-o://be68e25a2ebfc593faeba4a68857260ffb95d4f752dcf80a8b226cdda5af2d2a" gracePeriod=45
Jul 08 07:15:55.436795 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.436771    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:15:55.528363 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.528281    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776effdc-0b77-4070-8301-97e01a029d49-tls-certs\") pod \"776effdc-0b77-4070-8301-97e01a029d49\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") "
Jul 08 07:15:55.528363 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.528315    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-model-cache\") pod \"776effdc-0b77-4070-8301-97e01a029d49\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") "
Jul 08 07:15:55.528363 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.528340    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-home\") pod \"776effdc-0b77-4070-8301-97e01a029d49\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") "
Jul 08 07:15:55.528363 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.528364    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vm8l\" (UniqueName: \"kubernetes.io/projected/776effdc-0b77-4070-8301-97e01a029d49-kube-api-access-8vm8l\") pod \"776effdc-0b77-4070-8301-97e01a029d49\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") "
Jul 08 07:15:55.528678 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.528437    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-tmp-dir\") pod \"776effdc-0b77-4070-8301-97e01a029d49\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") "
Jul 08 07:15:55.528678 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.528484    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-dshm\") pod \"776effdc-0b77-4070-8301-97e01a029d49\" (UID: \"776effdc-0b77-4070-8301-97e01a029d49\") "
Jul 08 07:15:55.528678 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.528571    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-model-cache" (OuterVolumeSpecName: "model-cache") pod "776effdc-0b77-4070-8301-97e01a029d49" (UID: "776effdc-0b77-4070-8301-97e01a029d49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:15:55.528678 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.528584    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-home" (OuterVolumeSpecName: "home") pod "776effdc-0b77-4070-8301-97e01a029d49" (UID: "776effdc-0b77-4070-8301-97e01a029d49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:15:55.528894 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.528702    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "776effdc-0b77-4070-8301-97e01a029d49" (UID: "776effdc-0b77-4070-8301-97e01a029d49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:15:55.528894 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.528757    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:15:55.528894 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.528777    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:15:55.530404 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.530378    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-dshm" (OuterVolumeSpecName: "dshm") pod "776effdc-0b77-4070-8301-97e01a029d49" (UID: "776effdc-0b77-4070-8301-97e01a029d49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:15:55.530528 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.530504    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776effdc-0b77-4070-8301-97e01a029d49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "776effdc-0b77-4070-8301-97e01a029d49" (UID: "776effdc-0b77-4070-8301-97e01a029d49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:15:55.530528 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.530508    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776effdc-0b77-4070-8301-97e01a029d49-kube-api-access-8vm8l" (OuterVolumeSpecName: "kube-api-access-8vm8l") pod "776effdc-0b77-4070-8301-97e01a029d49" (UID: "776effdc-0b77-4070-8301-97e01a029d49"). InnerVolumeSpecName "kube-api-access-8vm8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:15:55.621083 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.621039    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" podUID="275c2425-5390-4987-804a-f70b52e9dd60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 08 07:15:55.629849 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.629826    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776effdc-0b77-4070-8301-97e01a029d49-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:15:55.629849 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.629845    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vm8l\" (UniqueName: \"kubernetes.io/projected/776effdc-0b77-4070-8301-97e01a029d49-kube-api-access-8vm8l\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:15:55.630094 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.629855    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:15:55.630094 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.629865    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776effdc-0b77-4070-8301-97e01a029d49-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:15:55.826977 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.826895    2582 generic.go:356] "Generic (PLEG): container finished" podID="776effdc-0b77-4070-8301-97e01a029d49" containerID="be68e25a2ebfc593faeba4a68857260ffb95d4f752dcf80a8b226cdda5af2d2a" exitCode=0
Jul 08 07:15:55.826977 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.826957    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"
Jul 08 07:15:55.827225 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.826976    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx" event={"ID":"776effdc-0b77-4070-8301-97e01a029d49","Type":"ContainerDied","Data":"be68e25a2ebfc593faeba4a68857260ffb95d4f752dcf80a8b226cdda5af2d2a"}
Jul 08 07:15:55.827225 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.827027    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx" event={"ID":"776effdc-0b77-4070-8301-97e01a029d49","Type":"ContainerDied","Data":"0552e02dd5d815ff6dc148b4ea5389f302a96b6d7a37d4f261e8df650d549ea1"}
Jul 08 07:15:55.827225 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.827048    2582 scope.go:117] "RemoveContainer" containerID="be68e25a2ebfc593faeba4a68857260ffb95d4f752dcf80a8b226cdda5af2d2a"
Jul 08 07:15:55.835960 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.835941    2582 scope.go:117] "RemoveContainer" containerID="be68e25a2ebfc593faeba4a68857260ffb95d4f752dcf80a8b226cdda5af2d2a"
Jul 08 07:15:55.836306 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:15:55.836281    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be68e25a2ebfc593faeba4a68857260ffb95d4f752dcf80a8b226cdda5af2d2a\": container with ID starting with be68e25a2ebfc593faeba4a68857260ffb95d4f752dcf80a8b226cdda5af2d2a not found: ID does not exist" containerID="be68e25a2ebfc593faeba4a68857260ffb95d4f752dcf80a8b226cdda5af2d2a"
Jul 08 07:15:55.836398 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.836316    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be68e25a2ebfc593faeba4a68857260ffb95d4f752dcf80a8b226cdda5af2d2a"} err="failed to get container status \"be68e25a2ebfc593faeba4a68857260ffb95d4f752dcf80a8b226cdda5af2d2a\": rpc error: code = NotFound desc = could not find container \"be68e25a2ebfc593faeba4a68857260ffb95d4f752dcf80a8b226cdda5af2d2a\": container with ID starting with be68e25a2ebfc593faeba4a68857260ffb95d4f752dcf80a8b226cdda5af2d2a not found: ID does not exist"
Jul 08 07:15:55.850197 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.850175    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"]
Jul 08 07:15:55.856281 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:55.856257    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98flcjrx"]
Jul 08 07:15:56.746751 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:15:56.746718    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776effdc-0b77-4070-8301-97e01a029d49" path="/var/lib/kubelet/pods/776effdc-0b77-4070-8301-97e01a029d49/volumes"
Jul 08 07:16:00.200867 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:00.200835    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:16:00.282495 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:00.282458    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:16:05.621224 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:05.621175    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" podUID="275c2425-5390-4987-804a-f70b52e9dd60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 08 07:16:15.621098 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:15.621046    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" podUID="275c2425-5390-4987-804a-f70b52e9dd60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 08 07:16:25.621475 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:25.621429    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" podUID="275c2425-5390-4987-804a-f70b52e9dd60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 08 07:16:26.581866 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.581828    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"]
Jul 08 07:16:26.582124 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.582113    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776effdc-0b77-4070-8301-97e01a029d49" containerName="main"
Jul 08 07:16:26.582195 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.582125    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="776effdc-0b77-4070-8301-97e01a029d49" containerName="main"
Jul 08 07:16:26.582232 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.582208    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="776effdc-0b77-4070-8301-97e01a029d49" containerName="main"
Jul 08 07:16:26.585385 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.585369    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.588309 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.588282    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 07:16:26.588432 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.588318    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 08 07:16:26.595985 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.595959    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"]
Jul 08 07:16:26.616101 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.616071    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:16:26.676315 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.676287    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.676315 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.676315    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.676732 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.676335    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.676732 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.676448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.676732 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.676491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.676732 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.676534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.676732 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.676550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.676732 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.676589    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqrvk\" (UniqueName: \"kubernetes.io/projected/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-kube-api-access-vqrvk\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.777552 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.777521    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.777552 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.777554    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.777751 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.777579    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.777751 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.777713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.777855 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.777776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.777855 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.777812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.777855 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.777835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.778003 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.777884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqrvk\" (UniqueName: \"kubernetes.io/projected/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-kube-api-access-vqrvk\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.778003 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.777930    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.778176 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.778131    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.778370 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.778335    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.778461 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.778245    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.778519 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.778495    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.780342 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.780317    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.780342 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.780329    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.786919 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.786872    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqrvk\" (UniqueName: \"kubernetes.io/projected/ec1d0b1a-7cd1-49d9-b280-796bd8f53eca-kube-api-access-vqrvk\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws\" (UID: \"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:26.840229 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.840141    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:16:26.896398 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:26.896367    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:27.040315 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:27.040130    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"]
Jul 08 07:16:27.042506 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:16:27.042470    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1d0b1a_7cd1_49d9_b280_796bd8f53eca.slice/crio-52bb8d2575ff3469e783884a8719e875898eadf1c139f2e1ad4d7c6ed523cdcf WatchSource:0}: Error finding container 52bb8d2575ff3469e783884a8719e875898eadf1c139f2e1ad4d7c6ed523cdcf: Status 404 returned error can't find the container with id 52bb8d2575ff3469e783884a8719e875898eadf1c139f2e1ad4d7c6ed523cdcf
Jul 08 07:16:27.973978 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:27.973938    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws" event={"ID":"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca","Type":"ContainerStarted","Data":"a4dab4dc745a8178aa2bbfb64122df941950699277a41f326a7c4af283354c57"}
Jul 08 07:16:27.974391 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:27.974007    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws" event={"ID":"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca","Type":"ContainerStarted","Data":"52bb8d2575ff3469e783884a8719e875898eadf1c139f2e1ad4d7c6ed523cdcf"}
Jul 08 07:16:31.995609 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:31.995570    2582 generic.go:356] "Generic (PLEG): container finished" podID="ec1d0b1a-7cd1-49d9-b280-796bd8f53eca" containerID="a4dab4dc745a8178aa2bbfb64122df941950699277a41f326a7c4af283354c57" exitCode=0
Jul 08 07:16:31.996026 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:31.995661    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws" event={"ID":"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca","Type":"ContainerDied","Data":"a4dab4dc745a8178aa2bbfb64122df941950699277a41f326a7c4af283354c57"}
Jul 08 07:16:33.003758 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:33.003713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws" event={"ID":"ec1d0b1a-7cd1-49d9-b280-796bd8f53eca","Type":"ContainerStarted","Data":"8389621bde1a4cda78271338d70f6ccf449aceaf5bb12758cc561b41ab3cee16"}
Jul 08 07:16:33.030698 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:33.030651    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws" podStartSLOduration=2.538038946 podStartE2EDuration="7.030635672s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.492596726s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:16:33.028966018 +0000 UTC m=+1308.832752420" watchObservedRunningTime="2026-07-08 07:16:33.030635672 +0000 UTC m=+1308.834422125"
Jul 08 07:16:33.897292 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:33.897255    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:34.882903 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:34.882866    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:16:35.621545 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:35.621496    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" podUID="275c2425-5390-4987-804a-f70b52e9dd60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 08 07:16:36.896896 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:36.896863    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:16:36.898337 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:36.898309    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws" podUID="ec1d0b1a-7cd1-49d9-b280-796bd8f53eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 08 07:16:45.621127 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:45.621075    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g" podUID="275c2425-5390-4987-804a-f70b52e9dd60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 08 07:16:46.897218 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:46.897133    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws" podUID="ec1d0b1a-7cd1-49d9-b280-796bd8f53eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 08 07:16:54.133825 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:54.133789    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerID="54ea71c8c9928bc406ca46baa093eb0032910d5a011465310c29efeb9e31882b" exitCode=137
Jul 08 07:16:54.134345 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:54.133859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerDied","Data":"54ea71c8c9928bc406ca46baa093eb0032910d5a011465310c29efeb9e31882b"}
Jul 08 07:16:54.134345 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:54.133926    2582 scope.go:117] "RemoveContainer" containerID="36c3a6e0cf7013208444f6739eb265949fe41b00dd4324d500c56c2c225d2206"
Jul 08 07:16:54.134345 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:54.134270    2582 scope.go:117] "RemoveContainer" containerID="54ea71c8c9928bc406ca46baa093eb0032910d5a011465310c29efeb9e31882b"
Jul 08 07:16:54.134522 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:16:54.134457    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:16:55.631127 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:55.631083    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:16:55.639707 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:55.639678    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-68f5d8959c-xh45g"
Jul 08 07:16:56.897227 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:16:56.897185    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws" podUID="ec1d0b1a-7cd1-49d9-b280-796bd8f53eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 08 07:17:01.636065 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:01.636018    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:17:01.636548 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:01.636529    2582 scope.go:117] "RemoveContainer" containerID="54ea71c8c9928bc406ca46baa093eb0032910d5a011465310c29efeb9e31882b"
Jul 08 07:17:01.636766 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:17:01.636741    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:17:06.897419 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:06.897375    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws" podUID="ec1d0b1a-7cd1-49d9-b280-796bd8f53eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 08 07:17:16.741313 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:16.741219    2582 scope.go:117] "RemoveContainer" containerID="54ea71c8c9928bc406ca46baa093eb0032910d5a011465310c29efeb9e31882b"
Jul 08 07:17:16.742330 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:16.742295    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:17:16.742453 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:16.742368    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:17:16.742453 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:16.742406    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:17:16.896821 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:16.896776    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws" podUID="ec1d0b1a-7cd1-49d9-b280-796bd8f53eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 08 07:17:17.242216 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:17.241336    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerStarted","Data":"e1aa6b1f99478041cc37480b4587e4168178a5cbcaf5bfb16f5458f8ca341368"}
Jul 08 07:17:17.515076 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:17.514990    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:17:17.516210 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:17.516184    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:17:17.516321 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:17.516238    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:17:18.515649 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:18.515610    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:17:18.516040 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:18.515666    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:17:19.515247 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:19.515209    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:17:19.515401 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:19.515265    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:17:20.515660 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:20.515607    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:17:20.516146 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:20.515685    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:17:21.515023 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:21.514981    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:17:21.515258 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:21.515047    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:17:22.515254 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:22.515209    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:17:22.515735 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:22.515272    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:17:24.515429 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:24.515379    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 07:17:24.515990 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:24.515448    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 07:17:24.934452 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:24.934418    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:17:24.934829 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:24.934792    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:17:24.935673 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:24.935650    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:17:26.896894 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:26.896848    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws" podUID="ec1d0b1a-7cd1-49d9-b280-796bd8f53eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 08 07:17:36.896849 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:36.896805    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws" podUID="ec1d0b1a-7cd1-49d9-b280-796bd8f53eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 08 07:17:42.374525 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:42.374485    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerID="e1aa6b1f99478041cc37480b4587e4168178a5cbcaf5bfb16f5458f8ca341368" exitCode=137
Jul 08 07:17:42.375038 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:42.374558    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerDied","Data":"e1aa6b1f99478041cc37480b4587e4168178a5cbcaf5bfb16f5458f8ca341368"}
Jul 08 07:17:42.375038 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:42.374637    2582 scope.go:117] "RemoveContainer" containerID="54ea71c8c9928bc406ca46baa093eb0032910d5a011465310c29efeb9e31882b"
Jul 08 07:17:42.375038 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:42.375012    2582 scope.go:117] "RemoveContainer" containerID="e1aa6b1f99478041cc37480b4587e4168178a5cbcaf5bfb16f5458f8ca341368"
Jul 08 07:17:42.375281 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:17:42.375258    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:17:46.897074 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:46.897021    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws" podUID="ec1d0b1a-7cd1-49d9-b280-796bd8f53eca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 08 07:17:54.935719 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:54.935687    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:17:54.936119 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:54.935997    2582 scope.go:117] "RemoveContainer" containerID="e1aa6b1f99478041cc37480b4587e4168178a5cbcaf5bfb16f5458f8ca341368"
Jul 08 07:17:54.936193 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:17:54.936168    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:17:56.745360 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:56.745328    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 07:17:56.753684 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:56.753653    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 07:17:56.778147 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:56.778124    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 07:17:56.795684 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:56.795664    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 07:17:56.907853 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:56.907826    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:17:56.915490 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:17:56.915468    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws"
Jul 08 07:18:03.929804 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:03.929766    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:18:04.264350 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:04.264260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:18:04.365020 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:04.364992    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:18:07.741175 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:07.741119    2582 scope.go:117] "RemoveContainer" containerID="e1aa6b1f99478041cc37480b4587e4168178a5cbcaf5bfb16f5458f8ca341368"
Jul 08 07:18:07.742240 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:07.742208    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:18:07.742347 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:07.742293    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:18:07.742411 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:07.742346    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:18:08.493067 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:08.493031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerStarted","Data":"0e0b2100ab90145f9abf8f51df2b1b252a32c0cc25d238f0f6090ef23cbb88f2"}
Jul 08 07:18:08.514748 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:08.514722    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:18:08.515761 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:08.515724    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:18:08.515901 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:08.515777    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:18:09.515251 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:09.515217    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:18:09.515630 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:09.515276    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:18:09.935007 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:09.934967    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:18:10.515361 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:10.515328    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:18:10.515743 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:10.515381    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:18:11.515107 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:11.515070    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:18:11.515301 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:11.515121    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:18:12.515068 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:12.515031    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:18:12.515482 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:12.515083    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:18:13.514949 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:13.514915    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:18:13.515228 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:13.514969    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:18:15.498564 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:15.498527    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:18:15.499900 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:15.499874    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:18:18.405369 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:18.405332    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:18:47.673233 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:47.673198    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerID="0e0b2100ab90145f9abf8f51df2b1b252a32c0cc25d238f0f6090ef23cbb88f2" exitCode=137
Jul 08 07:18:47.673669 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:47.673249    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerDied","Data":"0e0b2100ab90145f9abf8f51df2b1b252a32c0cc25d238f0f6090ef23cbb88f2"}
Jul 08 07:18:47.673669 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:47.673299    2582 scope.go:117] "RemoveContainer" containerID="e1aa6b1f99478041cc37480b4587e4168178a5cbcaf5bfb16f5458f8ca341368"
Jul 08 07:18:47.673669 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:47.673628    2582 scope.go:117] "RemoveContainer" containerID="0e0b2100ab90145f9abf8f51df2b1b252a32c0cc25d238f0f6090ef23cbb88f2"
Jul 08 07:18:47.673806 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:18:47.673784    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:18:58.745713 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:18:58.745686    2582 scope.go:117] "RemoveContainer" containerID="0e0b2100ab90145f9abf8f51df2b1b252a32c0cc25d238f0f6090ef23cbb88f2"
Jul 08 07:18:58.746093 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:18:58.745849    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:19:00.499668 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:00.499636    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:19:00.500099 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:00.500028    2582 scope.go:117] "RemoveContainer" containerID="0e0b2100ab90145f9abf8f51df2b1b252a32c0cc25d238f0f6090ef23cbb88f2"
Jul 08 07:19:00.500248 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:19:00.500227    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:19:14.742409 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:14.742378    2582 scope.go:117] "RemoveContainer" containerID="0e0b2100ab90145f9abf8f51df2b1b252a32c0cc25d238f0f6090ef23cbb88f2"
Jul 08 07:19:14.742792 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:19:14.742553    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:19:27.741293 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:27.741265    2582 scope.go:117] "RemoveContainer" containerID="0e0b2100ab90145f9abf8f51df2b1b252a32c0cc25d238f0f6090ef23cbb88f2"
Jul 08 07:19:27.742327 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:27.742297    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:19:27.742404 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:27.742365    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:19:27.742404 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:27.742398    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:19:27.842209 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:27.842176    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerStarted","Data":"4c3aa7bdc02c221f4f3372104ba4f44f849a5b06ee9ad0cae11eb0bebf146e21"}
Jul 08 07:19:28.514979 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:28.514944    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:19:28.515764 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:28.515734    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:19:28.515882 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:28.515784    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:19:29.514850 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:29.514809    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:19:29.515262 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:29.514894    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:19:30.499715 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:30.499678    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:19:30.515581 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:30.515556    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:19:30.515921 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:30.515602    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:19:31.515593 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:31.515558    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:19:31.515975 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:31.515615    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:19:32.514783 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:32.514749    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:19:32.514961 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:32.514804    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:19:33.514905 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:33.514870    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:19:33.515266 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:33.514922    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:19:35.448074 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:35.448044    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:19:35.449362 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:35.449340    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:19:44.713188 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:44.713140    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:19:44.713638 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:19:44.713623    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:20:07.019993 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:20:07.019966    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerID="4c3aa7bdc02c221f4f3372104ba4f44f849a5b06ee9ad0cae11eb0bebf146e21" exitCode=137
Jul 08 07:20:07.020345 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:20:07.020035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerDied","Data":"4c3aa7bdc02c221f4f3372104ba4f44f849a5b06ee9ad0cae11eb0bebf146e21"}
Jul 08 07:20:07.020345 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:20:07.020099    2582 scope.go:117] "RemoveContainer" containerID="0e0b2100ab90145f9abf8f51df2b1b252a32c0cc25d238f0f6090ef23cbb88f2"
Jul 08 07:20:07.020505 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:20:07.020488    2582 scope.go:117] "RemoveContainer" containerID="4c3aa7bdc02c221f4f3372104ba4f44f849a5b06ee9ad0cae11eb0bebf146e21"
Jul 08 07:20:07.020717 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:20:07.020694    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:20:17.741178 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:20:17.741130    2582 scope.go:117] "RemoveContainer" containerID="4c3aa7bdc02c221f4f3372104ba4f44f849a5b06ee9ad0cae11eb0bebf146e21"
Jul 08 07:20:17.741670 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:20:17.741318    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:20:20.448939 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:20:20.448901    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:20:20.449333 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:20:20.449296    2582 scope.go:117] "RemoveContainer" containerID="4c3aa7bdc02c221f4f3372104ba4f44f849a5b06ee9ad0cae11eb0bebf146e21"
Jul 08 07:20:20.449471 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:20:20.449451    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:20:34.581698 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:20:34.581661    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:20:34.745249 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:20:34.745222    2582 scope.go:117] "RemoveContainer" containerID="4c3aa7bdc02c221f4f3372104ba4f44f849a5b06ee9ad0cae11eb0bebf146e21"
Jul 08 07:20:34.745407 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:20:34.745389    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:20:47.741313 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:20:47.741284    2582 scope.go:117] "RemoveContainer" containerID="4c3aa7bdc02c221f4f3372104ba4f44f849a5b06ee9ad0cae11eb0bebf146e21"
Jul 08 07:20:47.741717 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:20:47.741474    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:20:53.663473 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:20:53.663435    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:20:59.740693 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:20:59.740659    2582 scope.go:117] "RemoveContainer" containerID="4c3aa7bdc02c221f4f3372104ba4f44f849a5b06ee9ad0cae11eb0bebf146e21"
Jul 08 07:20:59.741068 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:20:59.740831    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:21:02.031531 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.031496    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"]
Jul 08 07:21:02.034820 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.034803    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.040699 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.040673    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 08 07:21:02.056525 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.056503    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"]
Jul 08 07:21:02.106849 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.106825    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:21:02.188958 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.188936    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bcg\" (UniqueName: \"kubernetes.io/projected/eda082cc-9a14-4867-8ac5-19d540f8ec41-kube-api-access-d8bcg\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.189078 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.188967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eda082cc-9a14-4867-8ac5-19d540f8ec41-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.189078 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.188987    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.189078 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.189063    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.189227 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.189090    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.189227 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.189105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.189227 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.189122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.290311 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.290231    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.290311 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.290269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.290311 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.290285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.290311 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.290303    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.290627 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.290349    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bcg\" (UniqueName: \"kubernetes.io/projected/eda082cc-9a14-4867-8ac5-19d540f8ec41-kube-api-access-d8bcg\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.290627 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.290417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eda082cc-9a14-4867-8ac5-19d540f8ec41-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.290627 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.290446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.290798 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.290674    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.290937 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.290915    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.291062 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.291045    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.291292 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.291270    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.293457 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.293437    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eda082cc-9a14-4867-8ac5-19d540f8ec41-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.293653 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.293634    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eda082cc-9a14-4867-8ac5-19d540f8ec41-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.308634 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.308609    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bcg\" (UniqueName: \"kubernetes.io/projected/eda082cc-9a14-4867-8ac5-19d540f8ec41-kube-api-access-d8bcg\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz\" (UID: \"eda082cc-9a14-4867-8ac5-19d540f8ec41\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.347479 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.347452    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:02.353295 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.353271    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:21:02.490799 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:02.490753    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"]
Jul 08 07:21:02.493072 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:21:02.493043    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda082cc_9a14_4867_8ac5_19d540f8ec41.slice/crio-52c84f27d6af0dcf1417a9c8d801af9dc27695f7c931c026afc999d71a25accc WatchSource:0}: Error finding container 52c84f27d6af0dcf1417a9c8d801af9dc27695f7c931c026afc999d71a25accc: Status 404 returned error can't find the container with id 52c84f27d6af0dcf1417a9c8d801af9dc27695f7c931c026afc999d71a25accc
Jul 08 07:21:03.262079 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:03.262037    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" event={"ID":"eda082cc-9a14-4867-8ac5-19d540f8ec41","Type":"ContainerStarted","Data":"02994340f26e2c4e7b58a3efa3a339b0b764be9d8e2233a3aec5e9030cf0e9c5"}
Jul 08 07:21:03.262470 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:03.262097    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" event={"ID":"eda082cc-9a14-4867-8ac5-19d540f8ec41","Type":"ContainerStarted","Data":"52c84f27d6af0dcf1417a9c8d801af9dc27695f7c931c026afc999d71a25accc"}
Jul 08 07:21:07.281712 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:07.281677    2582 generic.go:356] "Generic (PLEG): container finished" podID="eda082cc-9a14-4867-8ac5-19d540f8ec41" containerID="02994340f26e2c4e7b58a3efa3a339b0b764be9d8e2233a3aec5e9030cf0e9c5" exitCode=0
Jul 08 07:21:07.282087 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:07.281746    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" event={"ID":"eda082cc-9a14-4867-8ac5-19d540f8ec41","Type":"ContainerDied","Data":"02994340f26e2c4e7b58a3efa3a339b0b764be9d8e2233a3aec5e9030cf0e9c5"}
Jul 08 07:21:08.288552 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:08.288515    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" event={"ID":"eda082cc-9a14-4867-8ac5-19d540f8ec41","Type":"ContainerStarted","Data":"840eb40da17cb7b8270fed3f09c1595c07b1f3ee221ab8de93103fe49dd89223"}
Jul 08 07:21:08.324744 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:08.324688    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" podStartSLOduration=3.215948616 podStartE2EDuration="7.324675949s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.108727333s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:21:08.321473958 +0000 UTC m=+1584.125260360" watchObservedRunningTime="2026-07-08 07:21:08.324675949 +0000 UTC m=+1584.128462349"
Jul 08 07:21:08.348376 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:08.348346    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:11.677045 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:11.677014    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:21:11.741097 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:11.741064    2582 scope.go:117] "RemoveContainer" containerID="4c3aa7bdc02c221f4f3372104ba4f44f849a5b06ee9ad0cae11eb0bebf146e21"
Jul 08 07:21:11.741367 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:21:11.741340    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:21:12.348245 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:12.348213    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:21:12.349844 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:12.349816    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" podUID="eda082cc-9a14-4867-8ac5-19d540f8ec41" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 08 07:21:22.348814 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:22.348770    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" podUID="eda082cc-9a14-4867-8ac5-19d540f8ec41" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 08 07:21:22.740598 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:22.740567    2582 scope.go:117] "RemoveContainer" containerID="4c3aa7bdc02c221f4f3372104ba4f44f849a5b06ee9ad0cae11eb0bebf146e21"
Jul 08 07:21:22.740779 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:21:22.740737    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:21:32.348479 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:32.348440    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" podUID="eda082cc-9a14-4867-8ac5-19d540f8ec41" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 08 07:21:37.740726 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:37.740647    2582 scope.go:117] "RemoveContainer" containerID="4c3aa7bdc02c221f4f3372104ba4f44f849a5b06ee9ad0cae11eb0bebf146e21"
Jul 08 07:21:37.741784 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:37.741752    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:21:37.741858 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:37.741842    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:21:37.741901 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:37.741888    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:21:38.429177 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:38.429113    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerStarted","Data":"24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"}
Jul 08 07:21:38.514681 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:38.514642    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:21:38.515665 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:38.515638    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:21:38.515784 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:38.515688    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:21:39.515256 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:39.515223    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:21:39.515633 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:39.515276    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:21:39.909570 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:39.909488    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:21:40.006343 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:40.006313    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:21:40.515254 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:40.515219    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:21:40.515425 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:40.515311    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:21:41.515354 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:41.515318    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:21:41.515536 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:41.515373    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:21:42.348183 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:42.348120    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" podUID="eda082cc-9a14-4867-8ac5-19d540f8ec41" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 08 07:21:42.515782 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:42.515738    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:21:42.516177 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:42.515805    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:21:43.515692 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:43.515658    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:21:43.515883 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:43.515720    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:21:45.515626 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:45.515583    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 07:21:45.516012 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:45.515648    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 07:21:46.190130 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:46.190088    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:21:46.190475 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:46.190452    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:21:46.191225 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:46.191203    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:21:52.348273 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:21:52.348232    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" podUID="eda082cc-9a14-4867-8ac5-19d540f8ec41" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 08 07:22:02.348290 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:02.348251    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" podUID="eda082cc-9a14-4867-8ac5-19d540f8ec41" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 08 07:22:09.600787 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:09.600757    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db" exitCode=137
Jul 08 07:22:09.601222 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:09.600820    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerDied","Data":"24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"}
Jul 08 07:22:09.601222 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:09.600868    2582 scope.go:117] "RemoveContainer" containerID="4c3aa7bdc02c221f4f3372104ba4f44f849a5b06ee9ad0cae11eb0bebf146e21"
Jul 08 07:22:09.601328 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:09.601278    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:22:09.601545 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:22:09.601507    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:22:12.348545 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:12.348491    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" podUID="eda082cc-9a14-4867-8ac5-19d540f8ec41" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 08 07:22:16.191120 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:16.191085    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:22:16.191594 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:16.191577    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:22:16.191796 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:22:16.191773    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:22:22.348253 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:22.348204    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" podUID="eda082cc-9a14-4867-8ac5-19d540f8ec41" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 08 07:22:27.741238 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:27.741210    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:22:27.741617 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:22:27.741371    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:22:32.347834 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:32.347773    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" podUID="eda082cc-9a14-4867-8ac5-19d540f8ec41" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 08 07:22:40.740867 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:40.740830    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:22:40.741354 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:22:40.741057    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:22:42.348234 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:42.348186    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" podUID="eda082cc-9a14-4867-8ac5-19d540f8ec41" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 08 07:22:52.348585 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:52.348542    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz" podUID="eda082cc-9a14-4867-8ac5-19d540f8ec41" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Jul 08 07:22:53.740379 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:22:53.740343    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:22:53.740843 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:22:53.740642    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:23:02.358557 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:23:02.358524    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:23:02.366930 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:23:02.366905    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz"
Jul 08 07:23:08.741192 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:23:08.741094    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:23:08.741530 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:23:08.741349    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:23:19.740532 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:23:19.740500    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:23:19.740999 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:23:19.740677    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:23:20.528986 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:23:20.528955    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:23:33.740461 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:23:33.740426    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:23:33.740906 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:23:33.740616    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:23:37.168668 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:23:37.168637    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:23:48.740712 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:23:48.740680    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:23:48.741195 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:23:48.740897    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:24:02.741335 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:02.741299    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:24:02.741808 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:24:02.741538    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:24:05.633790 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:05.633759    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:24:05.689410 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:05.689380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:24:16.740755 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:16.740719    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:24:16.741136 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:24:16.740915    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:24:29.741307 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:29.741271    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:24:29.741692 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:24:29.741457    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:24:44.739357 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:44.739327    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:24:44.742008 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:44.741983    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:24:44.745062 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:44.745035    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:24:44.745624 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:24:44.745592    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:24:55.740564 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:55.740533    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:24:55.741521 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:55.741489    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:24:55.741580 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:55.741558    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:24:55.741621 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:55.741587    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:24:56.324481 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:56.324441    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerStarted","Data":"0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"}
Jul 08 07:24:56.514958 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:56.514929    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:24:56.515857 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:56.515835    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:24:56.515930 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:56.515892    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:24:57.515036 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:57.514999    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:24:57.515441 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:57.515051    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:24:58.515652 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:58.515619    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:24:58.516052 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:58.515672    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:24:59.515476 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:59.515440    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:24:59.515657 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:24:59.515508    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:25:00.515031 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:00.514994    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:25:00.515443 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:00.515050    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:25:01.191063 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:01.191015    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:25:01.515738 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:01.515651    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:25:01.515738 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:01.515711    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:25:03.516699 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:03.516654    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 07:25:03.517132 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:03.516725    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 07:25:04.195330 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:04.195297    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:25:04.196566 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:04.196543    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:25:28.483870 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:28.483829    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b" exitCode=137
Jul 08 07:25:28.484340 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:28.483896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerDied","Data":"0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"}
Jul 08 07:25:28.484340 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:28.483960    2582 scope.go:117] "RemoveContainer" containerID="24f4c3235a0ebe795d4769d94a6a54dbac0d68ed8ac15954eabfc34d850624db"
Jul 08 07:25:28.484435 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:28.484384    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:25:28.484591 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:25:28.484562    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:25:34.196068 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:34.196032    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:25:34.196482 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:34.196421    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:25:34.196592 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:25:34.196573    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:25:46.740807 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:46.740776    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:25:46.741194 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:25:46.740992    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:25:57.740899 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:25:57.740873    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:25:57.741287 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:25:57.741035    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:26:10.740610 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:26:10.740538    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:26:10.741069 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:26:10.740704    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:26:23.740349 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:26:23.740316    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:26:23.740788 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:26:23.740550    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:26:35.740900 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:26:35.740873    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:26:35.741355 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:26:35.741097    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:26:49.740833 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:26:49.740799    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:26:49.741349 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:26:49.740963    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:27:01.741022 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:27:01.740989    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:27:01.741416 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:27:01.741184    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:27:14.743444 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:27:14.743414    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:27:14.743832 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:27:14.743576    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:27:25.741382 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:27:25.741354    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:27:25.742069 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:27:25.741516    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:27:36.740967 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:27:36.740935    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:27:36.741375 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:27:36.741098    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:27:48.740944 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:27:48.740912    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:27:48.741338 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:27:48.741094    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:27:49.317564 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:27:49.317530    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:27:50.746273 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:27:50.746245    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:28:01.740854 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:28:01.740821    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:28:01.741359 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:28:01.740991    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:28:13.740726 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:28:13.740695    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:28:13.741110 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:28:13.740881    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:28:27.741362 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:28:27.741330    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:28:27.743668 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:28:27.741505    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:28:42.740617 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:28:42.740582    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:28:42.741113 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:28:42.740771    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:28:56.741211 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:28:56.741173    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:28:56.741609 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:28:56.741379    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:29:09.741411 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:29:09.741325    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:29:09.741825 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:29:09.741511    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:29:17.655026 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:29:17.654978    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:29:22.741416 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:29:22.741387    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:29:22.741803 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:29:22.741576    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:29:34.742562 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:29:34.742532    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:29:34.742946 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:29:34.742718    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:29:44.771871 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:29:44.771842    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:29:44.775827 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:29:44.775804    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:29:47.741034 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:29:47.741008    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:29:47.741459 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:29:47.741179    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:30:01.740434 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:01.740406    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:30:01.740897 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:30:01.740566    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:30:14.745530 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:14.745500    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:30:14.746023 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:30:14.745676    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:30:28.740644 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:28.740612    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:30:28.741690 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:28.741656    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:30:28.741770 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:28.741742    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:30:28.741831 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:28.741808    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:30:29.773790 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:29.773759    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerStarted","Data":"27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"}
Jul 08 07:30:30.515370 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:30.515329    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:30:30.516456 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:30.516433    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:30:30.516518 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:30.516488    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:30:31.515574 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:31.515536    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:30:31.515984 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:31.515617    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:30:32.515394 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:32.515357    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:30:32.515584 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:32.515423    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:30:33.514827 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:33.514781    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:30:33.515315 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:33.514846    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:30:34.196416 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:34.196367    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:30:34.515002 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:34.514915    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:30:34.515002 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:34.514969    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:30:35.515802 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:35.515756    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:30:35.516218 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:35.515824    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:30:37.516776 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:37.516677    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 07:30:37.516776 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:37.516758    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 07:30:38.178247 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:38.178209    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:30:38.179521 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:30:38.179483    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:31:01.919564 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:31:01.919529    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339" exitCode=137
Jul 08 07:31:01.920120 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:31:01.919579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerDied","Data":"27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"}
Jul 08 07:31:01.920120 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:31:01.919625    2582 scope.go:117] "RemoveContainer" containerID="0bc9203528b8153716adb2e7a9ec820ece42fcd8f36303ca1afaab0d38f3539b"
Jul 08 07:31:01.920120 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:31:01.920063    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:31:01.920350 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:31:01.920323    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:31:08.179184 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:31:08.179120    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:31:08.179705 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:31:08.179688    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:31:08.179950 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:31:08.179925    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:31:20.741282 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:31:20.741245    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:31:20.741670 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:31:20.741414    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:31:35.740670 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:31:35.740638    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:31:35.741198 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:31:35.740809    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:31:47.741403 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:31:47.741370    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:31:47.741809 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:31:47.741579    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:32:00.741087 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:32:00.741050    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:32:00.741569 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:32:00.741275    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:32:15.740668 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:32:15.740580    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:32:15.741201 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:32:15.740813    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:32:30.740807 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:32:30.740773    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:32:30.741207 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:32:30.741009    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:32:41.740592 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:32:41.740565    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:32:41.740966 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:32:41.740744    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:32:52.740741 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:32:52.740713    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:32:52.741236 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:32:52.740897    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:33:03.020220 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:03.020181    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 07:33:03.029206 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:03.029178    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 07:33:06.741436 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:06.741397    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:33:06.741896 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:33:06.741646    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:33:12.499836 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.499799    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"]
Jul 08 07:33:12.503023 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.502998    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.505586 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.505564    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 08 07:33:12.520383 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.520359    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"]
Jul 08 07:33:12.600390 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.600359    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:33:12.608629 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.608603    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.608761 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.608637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57b1e783-1750-4f22-8ebd-bc305536de13-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.608761 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.608665    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.608761 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.608683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.608761 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.608731    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48v8h\" (UniqueName: \"kubernetes.io/projected/57b1e783-1750-4f22-8ebd-bc305536de13-kube-api-access-48v8h\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.608761 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.608753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.608931 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.608796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.608931 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.608825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57b1e783-1750-4f22-8ebd-bc305536de13-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.709597 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.709565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48v8h\" (UniqueName: \"kubernetes.io/projected/57b1e783-1750-4f22-8ebd-bc305536de13-kube-api-access-48v8h\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.709597 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.709600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.709787 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.709631    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.709787 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.709646    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57b1e783-1750-4f22-8ebd-bc305536de13-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.709787 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.709701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.709787 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.709731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57b1e783-1750-4f22-8ebd-bc305536de13-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.709787 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.709772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.710045 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.709793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.710136 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.710114    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.710214 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.710195    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.710263 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.710233    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.710401 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.710381    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.710645 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.710628    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57b1e783-1750-4f22-8ebd-bc305536de13-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.711841 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.711826    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57b1e783-1750-4f22-8ebd-bc305536de13-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.712327 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.712304    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57b1e783-1750-4f22-8ebd-bc305536de13-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.718484 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.718456    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48v8h\" (UniqueName: \"kubernetes.io/projected/57b1e783-1750-4f22-8ebd-bc305536de13-kube-api-access-48v8h\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6\" (UID: \"57b1e783-1750-4f22-8ebd-bc305536de13\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.814402 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.814329    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:12.952192 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.952161    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"]
Jul 08 07:33:12.954352 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:33:12.954312    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b1e783_1750_4f22_8ebd_bc305536de13.slice/crio-0b9b603c1f3b90604e37905af691ae6d0f21000a3ca98f0e8dcc7a219b862e82 WatchSource:0}: Error finding container 0b9b603c1f3b90604e37905af691ae6d0f21000a3ca98f0e8dcc7a219b862e82: Status 404 returned error can't find the container with id 0b9b603c1f3b90604e37905af691ae6d0f21000a3ca98f0e8dcc7a219b862e82
Jul 08 07:33:12.959441 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:12.959420    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 07:33:13.480282 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:13.480241    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6" event={"ID":"57b1e783-1750-4f22-8ebd-bc305536de13","Type":"ContainerStarted","Data":"65a1fb2b67a4d58bc0471ddb72ac701aaecb0b2070d06a1afed2c83a6624de9f"}
Jul 08 07:33:13.480487 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:13.480296    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6" event={"ID":"57b1e783-1750-4f22-8ebd-bc305536de13","Type":"ContainerStarted","Data":"0b9b603c1f3b90604e37905af691ae6d0f21000a3ca98f0e8dcc7a219b862e82"}
Jul 08 07:33:14.064121 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:14.064092    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:33:17.502705 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:17.502667    2582 generic.go:356] "Generic (PLEG): container finished" podID="57b1e783-1750-4f22-8ebd-bc305536de13" containerID="65a1fb2b67a4d58bc0471ddb72ac701aaecb0b2070d06a1afed2c83a6624de9f" exitCode=0
Jul 08 07:33:17.503236 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:17.502746    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6" event={"ID":"57b1e783-1750-4f22-8ebd-bc305536de13","Type":"ContainerDied","Data":"65a1fb2b67a4d58bc0471ddb72ac701aaecb0b2070d06a1afed2c83a6624de9f"}
Jul 08 07:33:17.740654 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:17.740625    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:33:17.740824 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:33:17.740777    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:33:18.509928 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:18.509896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6" event={"ID":"57b1e783-1750-4f22-8ebd-bc305536de13","Type":"ContainerStarted","Data":"069334125f3f3bed0010fe6043f1b884e6ca3f3f34b913a48f9e8cffb8376206"}
Jul 08 07:33:18.534511 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:18.534452    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6" podStartSLOduration=2.659310728 podStartE2EDuration="6.534433825s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.875123097s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:33:18.533195103 +0000 UTC m=+2314.336981524" watchObservedRunningTime="2026-07-08 07:33:18.534433825 +0000 UTC m=+2314.338220226"
Jul 08 07:33:18.815462 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:18.815381    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:22.815522 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:22.815437    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:33:22.817128 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:22.817101    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6" podUID="57b1e783-1750-4f22-8ebd-bc305536de13" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 08 07:33:23.664118 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:23.664080    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:33:29.740992 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:29.740955    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:33:29.742465 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:33:29.741145    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:33:32.815313 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:32.815269    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6" podUID="57b1e783-1750-4f22-8ebd-bc305536de13" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 08 07:33:41.741418 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:41.741338    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:33:41.741861 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:33:41.741540    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:33:42.815525 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:42.815471    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6" podUID="57b1e783-1750-4f22-8ebd-bc305536de13" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 08 07:33:52.814747 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:52.814697    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6" podUID="57b1e783-1750-4f22-8ebd-bc305536de13" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 08 07:33:54.742441 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:33:54.742411    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:33:54.742861 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:33:54.742597    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:34:02.815406 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:34:02.815360    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6" podUID="57b1e783-1750-4f22-8ebd-bc305536de13" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 08 07:34:09.740698 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:34:09.740668    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:34:09.741130 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:34:09.740901    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:34:12.815318 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:34:12.815270    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6" podUID="57b1e783-1750-4f22-8ebd-bc305536de13" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 08 07:34:20.740880 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:34:20.740842    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:34:20.741282 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:34:20.741015    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:34:22.815809 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:34:22.815715    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6" podUID="57b1e783-1750-4f22-8ebd-bc305536de13" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 08 07:34:32.825673 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:34:32.825641    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:34:32.834029 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:34:32.834001    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6"
Jul 08 07:34:35.740957 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:34:35.740927    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:34:35.741368 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:34:35.741105    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:34:44.799272 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:34:44.799245    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:34:44.807611 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:34:44.807583    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:34:47.741032 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:34:47.741000    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:34:47.741447 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:34:47.741221    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:34:59.741351 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:34:59.741319    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:34:59.741748 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:34:59.741499    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:35:14.742922 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:35:14.742828    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:35:14.743315 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:35:14.743077    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:35:26.741084 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:35:26.741049    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:35:26.741570 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:35:26.741329    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:35:37.741040 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:35:37.741009    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:35:37.741541 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:35:37.741204    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:35:42.537421 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:35:42.537378    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:35:42.606809 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:35:42.606761    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:35:46.284875 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:35:46.284841    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:35:52.741518 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:35:52.741482    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:35:52.741998 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:35:52.741728    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:36:03.741146 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:03.741115    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:36:03.742232 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:03.742195    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:36:03.742355 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:03.742268    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:36:03.742355 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:03.742302    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:36:04.283726 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:04.283685    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerStarted","Data":"2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"}
Jul 08 07:36:04.515442 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:04.515407    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:36:04.516442 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:04.516412    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:36:04.516565 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:04.516465    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:36:05.515716 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:05.515682    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:36:05.516223 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:05.515742    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:36:06.515327 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:06.515290    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:36:06.515513 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:06.515345    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:36:07.514919 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:07.514883    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:36:07.515346 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:07.514971    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:36:08.179350 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:08.179316    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:36:08.515742 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:08.515648    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:36:08.515742 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:08.515723    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:36:09.515277 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:09.515240    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:36:09.515537 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:09.515296    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:36:11.556359 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:11.556310    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 07:36:11.556854 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:11.556384    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 07:36:12.305608 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:12.305576    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:36:12.306849 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:12.306815    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:36:38.497640 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:38.497560    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 07:36:38.507494 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:38.507468    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 07:36:39.735021 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:39.734989    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:36:39.835990 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:39.835957    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:36:39.890202 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:39.890173    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:36:41.468810 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:41.468776    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e" exitCode=137
Jul 08 07:36:41.469310 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:41.468846    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerDied","Data":"2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"}
Jul 08 07:36:41.469310 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:41.468914    2582 scope.go:117] "RemoveContainer" containerID="27b695f57d1b4b3efd040727bd40f4bf769483497387367cc89c1cb8679ea339"
Jul 08 07:36:41.469406 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:41.469348    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:36:41.469596 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:36:41.469573    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:36:42.292189 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:42.292141    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:36:42.306685 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:42.306653    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:36:42.391781 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:42.391740    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:36:42.478719 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:42.478687    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:36:42.479215 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:36:42.478929    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:36:42.889364 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:42.889328    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:36:53.740511 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:36:53.740478    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:36:53.740931 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:36:53.740668    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:37:05.740762 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:05.740728    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:37:05.741184 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:37:05.740912    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:37:16.740534 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:16.740502    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:37:16.741029 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:37:16.740697    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:37:30.741243 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:30.741211    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:37:30.741635 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:37:30.741397    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:37:33.595609 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.595572    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"]
Jul 08 07:37:33.598878 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.598857    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.601739 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.601717    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 08 07:37:33.612820 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.612793    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"]
Jul 08 07:37:33.634252 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.634217    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:37:33.724971 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.724938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.725194 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.724984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.725194 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.725035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.725194 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.725161    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.725343 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.725203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.725343 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.725229    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghj78\" (UniqueName: \"kubernetes.io/projected/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-kube-api-access-ghj78\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.826246 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.826210    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.826449 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.826314    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.826449 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.826341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.826449 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.826374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghj78\" (UniqueName: \"kubernetes.io/projected/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-kube-api-access-ghj78\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.826449 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.826416    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.826667 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.826457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.826742 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.826714    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.826920 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.826890    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.827029 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.826931    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.828699 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.828666    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.828976 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.828956    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.838836 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.838812    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghj78\" (UniqueName: \"kubernetes.io/projected/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-kube-api-access-ghj78\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:33.884726 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:33.884638    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:37:36.857139 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:36.857095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:36.863119 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:36.863086    2582 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:36.958572 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:36.958519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:36.958733 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:36.958633    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:36.961446 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:36.961423    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") DevicePath \"csi-39614a8fa7302f6a5e404233f1c9304664ec83103aea2cf5242a35c67533acd5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:37.055762 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:37.055727    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d8cc943a74105dc3dfe7dcd9ae9ba70ab2d5536dabcf5b22935bc499f06ab79b/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:37.072204 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:37.072145    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:37.210721 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:37.210679    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:37.361585 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:37.361548    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"]
Jul 08 07:37:37.364463 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:37:37.364428    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29043fbe_c2ca_48bc_a0a9_9fb15b75fac4.slice/crio-c9c99e758d0d78de4bb2567da5280f013b162989dfad5a122589585b17ed3e7d WatchSource:0}: Error finding container c9c99e758d0d78de4bb2567da5280f013b162989dfad5a122589585b17ed3e7d: Status 404 returned error can't find the container with id c9c99e758d0d78de4bb2567da5280f013b162989dfad5a122589585b17ed3e7d
Jul 08 07:37:37.723781 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:37.723741    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" event={"ID":"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4","Type":"ContainerStarted","Data":"d4f828506edbebce64d821529fb4236eb72089f646e8c95d1757d90871309010"}
Jul 08 07:37:37.723970 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:37.723819    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" event={"ID":"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4","Type":"ContainerStarted","Data":"c9c99e758d0d78de4bb2567da5280f013b162989dfad5a122589585b17ed3e7d"}
Jul 08 07:37:37.745911 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:37.745843    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podStartSLOduration=4.745822737 podStartE2EDuration="4.745822737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 07:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:37:37.744503065 +0000 UTC m=+2573.548289487" watchObservedRunningTime="2026-07-08 07:37:37.745822737 +0000 UTC m=+2573.549609130"
Jul 08 07:37:38.211270 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:38.211231    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:42.740879 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:42.740840    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:37:42.741403 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:37:42.741017    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:37:45.731099 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:45.731062    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:37:47.211427 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:47.211389    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:37:47.212922 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:47.212889    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 08 07:37:56.741344 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:56.741311    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:37:56.741828 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:37:56.741504    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:37:57.211594 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:37:57.211551    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 08 07:38:07.211567 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:07.211517    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 08 07:38:08.314167 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:08.314113    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 07:38:08.323056 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:08.323020    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 07:38:11.741199 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:11.741165    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:38:11.741628 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:38:11.741351    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:38:17.211408 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:17.211357    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 08 07:38:22.005510 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.005470    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"]
Jul 08 07:38:22.009209 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.009186    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.012019 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.011996    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 08 07:38:22.023256 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.023231    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"]
Jul 08 07:38:22.031614 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.031589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:38:22.080565 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.080523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-tmp-dir\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.080752 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.080581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-dshm\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.080752 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.080657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-model-cache\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.080752 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.080692    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552f4e64-1b67-44ee-adf4-56fdad788beb-tls-certs\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.080752 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.080733    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql2hm\" (UniqueName: \"kubernetes.io/projected/552f4e64-1b67-44ee-adf4-56fdad788beb-kube-api-access-ql2hm\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.080923 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.080807    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-home\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.080923 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.080837    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.181390 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.181359    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-dshm\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.181601 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.181409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-model-cache\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.181601 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.181429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552f4e64-1b67-44ee-adf4-56fdad788beb-tls-certs\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.181601 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.181460    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql2hm\" (UniqueName: \"kubernetes.io/projected/552f4e64-1b67-44ee-adf4-56fdad788beb-kube-api-access-ql2hm\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.181601 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.181486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-home\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.181601 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.181506    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.181601 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.181563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-tmp-dir\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.181930 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.181838    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-model-cache\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.181930 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.181892    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-home\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.182210 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.182186    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.182313 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.182217    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-tmp-dir\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.183786 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.183755    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552f4e64-1b67-44ee-adf4-56fdad788beb-dshm\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.184050 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.184031    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552f4e64-1b67-44ee-adf4-56fdad788beb-tls-certs\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.189546 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.189513    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql2hm\" (UniqueName: \"kubernetes.io/projected/552f4e64-1b67-44ee-adf4-56fdad788beb-kube-api-access-ql2hm\") pod \"custom-route-timeout-test-kserve-b6f9dd5f7-llmnq\" (UID: \"552f4e64-1b67-44ee-adf4-56fdad788beb\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.319530 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.319435    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:22.473777 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.473745    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"]
Jul 08 07:38:22.476218 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:38:22.476184    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552f4e64_1b67_44ee_adf4_56fdad788beb.slice/crio-2f5100e008650caf13da813d3c1ea465281b190e7c9d2e4a27621ce19b5455fb WatchSource:0}: Error finding container 2f5100e008650caf13da813d3c1ea465281b190e7c9d2e4a27621ce19b5455fb: Status 404 returned error can't find the container with id 2f5100e008650caf13da813d3c1ea465281b190e7c9d2e4a27621ce19b5455fb
Jul 08 07:38:22.740759 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.740729    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:38:22.740956 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:38:22.740933    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:38:22.949342 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.949286    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" event={"ID":"552f4e64-1b67-44ee-adf4-56fdad788beb","Type":"ContainerStarted","Data":"b5713160351417943dd5e98f001ad4028a78b4e0315728bfb8f2eab91137feb3"}
Jul 08 07:38:22.949520 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:22.949367    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" event={"ID":"552f4e64-1b67-44ee-adf4-56fdad788beb","Type":"ContainerStarted","Data":"2f5100e008650caf13da813d3c1ea465281b190e7c9d2e4a27621ce19b5455fb"}
Jul 08 07:38:23.196941 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:23.196907    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:38:26.977757 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:26.977713    2582 generic.go:356] "Generic (PLEG): container finished" podID="552f4e64-1b67-44ee-adf4-56fdad788beb" containerID="b5713160351417943dd5e98f001ad4028a78b4e0315728bfb8f2eab91137feb3" exitCode=0
Jul 08 07:38:26.978193 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:26.977759    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" event={"ID":"552f4e64-1b67-44ee-adf4-56fdad788beb","Type":"ContainerDied","Data":"b5713160351417943dd5e98f001ad4028a78b4e0315728bfb8f2eab91137feb3"}
Jul 08 07:38:27.211875 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:27.211828    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 08 07:38:27.984896 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:27.984850    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" event={"ID":"552f4e64-1b67-44ee-adf4-56fdad788beb","Type":"ContainerStarted","Data":"a04c5a372d0f050d217a4d7c20ac1caef2e8ff526ed1d5582686b1f0b81ed5fa"}
Jul 08 07:38:28.012050 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:28.012002    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" podStartSLOduration=3.328625957 podStartE2EDuration="7.011986302s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.683360345s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:38:28.011006228 +0000 UTC m=+2623.814792644" watchObservedRunningTime="2026-07-08 07:38:28.011986302 +0000 UTC m=+2623.815772701"
Jul 08 07:38:28.320371 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:28.320281    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:32.319885 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:32.319841    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:38:32.322464 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:32.321350    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" podUID="552f4e64-1b67-44ee-adf4-56fdad788beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 08 07:38:34.245795 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:34.245746    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:38:35.740494 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:35.740456    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:38:35.740992 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:38:35.740655    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:38:37.211370 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:37.211315    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 08 07:38:42.319845 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:42.319771    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" podUID="552f4e64-1b67-44ee-adf4-56fdad788beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 08 07:38:46.740537 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:46.740497    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:38:46.741051 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:38:46.740712    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:38:47.211976 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:47.211912    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 08 07:38:52.320519 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:52.320470    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" podUID="552f4e64-1b67-44ee-adf4-56fdad788beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 08 07:38:57.211381 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:57.211328    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 08 07:38:59.741258 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:38:59.741220    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:38:59.741759 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:38:59.741420    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:39:02.320076 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:02.320028    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" podUID="552f4e64-1b67-44ee-adf4-56fdad788beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 08 07:39:07.211764 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:07.211713    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 08 07:39:12.320530 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:12.320481    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" podUID="552f4e64-1b67-44ee-adf4-56fdad788beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 08 07:39:14.742677 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:14.742646    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:39:14.743313 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:39:14.742856    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:39:17.211482 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:17.211429    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 08 07:39:22.320113 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:22.320065    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" podUID="552f4e64-1b67-44ee-adf4-56fdad788beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 08 07:39:27.211096 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:27.211035    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 08 07:39:29.741000 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:29.740961    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:39:29.741422 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:39:29.741241    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:39:32.320526 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:32.320476    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" podUID="552f4e64-1b67-44ee-adf4-56fdad788beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 08 07:39:37.211970 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:37.211922    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 08 07:39:42.320705 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:42.320657    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" podUID="552f4e64-1b67-44ee-adf4-56fdad788beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 08 07:39:44.742787 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:44.742744    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:39:44.743297 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:39:44.743037    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:39:44.845482 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:44.845449    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:39:44.848176 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:44.848128    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:39:47.221184 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:47.221136    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:39:47.229533 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:47.229503    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:39:52.320524 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:52.320481    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" podUID="552f4e64-1b67-44ee-adf4-56fdad788beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 08 07:39:57.740415 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:39:57.740375    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:39:57.740921 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:39:57.740559    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:40:02.320554 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:40:02.320506    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" podUID="552f4e64-1b67-44ee-adf4-56fdad788beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 08 07:40:06.302754 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:40:06.302711    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:40:09.741370 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:40:09.741335    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:40:09.741749 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:40:09.741551    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:40:12.320757 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:40:12.320695    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" podUID="552f4e64-1b67-44ee-adf4-56fdad788beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 08 07:40:21.740708 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:40:21.740667    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:40:21.741179 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:40:21.740925    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:40:22.320845 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:40:22.320796    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq" podUID="552f4e64-1b67-44ee-adf4-56fdad788beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 08 07:40:32.330874 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:40:32.330843    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:40:32.339582 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:40:32.339554    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-b6f9dd5f7-llmnq"
Jul 08 07:40:36.741310 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:40:36.741276    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:40:36.741782 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:40:36.741512    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:40:47.741141 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:40:47.741107    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:40:47.741553 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:40:47.741324    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:41:00.548859 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:00.548821    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:41:00.611909 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:00.611883    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:41:01.740304 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:01.740265    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:41:01.740682 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:41:01.740460    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:41:16.741364 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:16.741292    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:41:16.741720 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:41:16.741503    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:41:31.740652 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:31.740622    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:41:31.741238 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:41:31.740791    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:41:45.741173 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:45.741130    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:41:45.742284 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:45.742252    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:41:45.742359 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:45.742325    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:41:45.742396 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:45.742363    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:41:45.991681 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:45.991596    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerStarted","Data":"1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"}
Jul 08 07:41:46.515545 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:46.515507    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:41:46.516262 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:46.516236    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:41:46.516373 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:46.516291    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:41:47.515371 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:47.515336    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:41:47.515757 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:47.515393    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:41:48.515827 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:48.515779    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:41:48.516234 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:48.515856    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:41:49.515398 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:49.515360    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:41:49.515585 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:49.515423    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:41:50.515744 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:50.515706    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:41:50.516191 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:50.515765    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:41:51.515258 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:51.515222    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:41:51.515434 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:51.515289    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:41:53.515706 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:53.515661    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 07:41:53.516276 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:53.515732    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 07:41:54.557334 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:54.557294    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 07:41:54.557860 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:54.557362    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 07:41:54.665965 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:54.665931    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:41:54.666493 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:54.666470    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:41:54.667303 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:41:54.667279    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:42:36.241596 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:42:36.241560    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c" exitCode=137
Jul 08 07:42:36.242092 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:42:36.241628    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerDied","Data":"1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"}
Jul 08 07:42:36.242092 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:42:36.241689    2582 scope.go:117] "RemoveContainer" containerID="2a869677467b3a5dc43bc18774187111af6937ca40f0107653814aaaed90328e"
Jul 08 07:42:36.242092 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:42:36.242043    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:42:36.242281 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:42:36.242261    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:42:39.666758 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:42:39.666670    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:42:39.667249 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:42:39.667199    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:42:39.667458 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:42:39.667433    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:42:52.740693 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:42:52.740663    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:42:52.741214 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:42:52.740872    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:43:04.743806 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:43:04.743760    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:43:04.744354 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:43:04.743957    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:43:16.740911 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:43:16.740874    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:43:16.741406 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:43:16.741096    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:43:27.740725 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:43:27.740695    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:43:27.741112 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:43:27.740875    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:43:41.740599 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:43:41.740564    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:43:41.742947 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:43:41.740758    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:43:56.741494 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:43:56.741459    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:43:56.742105 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:43:56.741638    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:44:10.740851 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:44:10.740764    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:44:10.741337 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:44:10.740938    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:44:21.740854 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:44:21.740819    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:44:21.741252 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:44:21.741012    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:44:33.740711 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:44:33.740679    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:44:33.741212 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:44:33.740872    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:44:44.745557 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:44:44.745500    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:44:44.746028 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:44:44.745889    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:44:44.872688 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:44:44.872660    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:44:44.877868 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:44:44.877840    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:44:58.741042 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:44:58.741016    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:44:58.741456 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:44:58.741181    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:45:06.946411 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:45:06.946374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:45:10.751387 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:45:10.751354    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:45:10.751801 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:45:10.751574    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:45:25.740810 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:45:25.740771    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:45:25.741216 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:45:25.740944    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:45:40.740685 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:45:40.740594    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:45:40.741248 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:45:40.740841    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:45:52.741436 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:45:52.741406    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:45:52.741966 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:45:52.741652    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:46:07.740894 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:46:07.740863    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:46:07.741386 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:46:07.741047    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:46:21.740757 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:46:21.740723    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:46:21.741279 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:46:21.740971    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:46:34.742835 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:46:34.742796    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:46:34.743240 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:46:34.743043    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:46:45.740941 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:46:45.740905    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:46:45.741430 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:46:45.741085    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:46:56.740629 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:46:56.740592    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:46:56.741025 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:46:56.740829    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:47:08.740822 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:08.740746    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:47:08.741229 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:47:08.740930    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:47:21.741173 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:21.741126    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:47:21.741693 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:47:21.741333    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:47:36.740871 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:36.740840    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:47:36.741974 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:36.741939    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:47:36.742045 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:36.742027    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:47:36.742094 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:36.742079    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:47:37.657547 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:37.657506    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerStarted","Data":"dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"}
Jul 08 07:47:38.514976 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:38.514940    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:47:38.516114 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:38.516085    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:47:38.516263 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:38.516145    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:47:39.514783 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:39.514747    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:47:39.514996 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:39.514804    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:47:39.666830 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:39.666800    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:47:40.515779 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:40.515747    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:47:40.516181 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:40.515831    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:47:41.515482 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:41.515450    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:47:41.515693 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:41.515504    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:47:42.515582 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:42.515547    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:47:42.516084 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:42.515599    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:47:44.516356 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:44.516317    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 07:47:44.516836 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:44.516375    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 07:47:45.558329 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:45.558289    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 07:47:45.558828 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:45.558361    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 07:47:45.655861 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:45.655823    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:47:45.668339 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:45.668313    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:47:57.411964 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:57.411921    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"]
Jul 08 07:47:57.429653 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:57.429623    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:47:57.502671 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:47:57.502634    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:48:12.439191 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:12.439118    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main" containerID="cri-o://d4f828506edbebce64d821529fb4236eb72089f646e8c95d1757d90871309010" gracePeriod=45
Jul 08 07:48:13.255923 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.255889    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:48:13.381510 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.381432    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-home\") pod \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") "
Jul 08 07:48:13.381664 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.381524    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-dshm\") pod \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") "
Jul 08 07:48:13.381664 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.381575    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-tls-certs\") pod \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") "
Jul 08 07:48:13.381664 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.381608    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-tmp-dir\") pod \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") "
Jul 08 07:48:13.381664 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.381638    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-model-cache\") pod \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") "
Jul 08 07:48:13.381886 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.381667    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghj78\" (UniqueName: \"kubernetes.io/projected/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-kube-api-access-ghj78\") pod \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") "
Jul 08 07:48:13.381886 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.381765    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\" (UID: \"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4\") "
Jul 08 07:48:13.381996 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.381920    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-model-cache" (OuterVolumeSpecName: "model-cache") pod "29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" (UID: "29043fbe-c2ca-48bc-a0a9-9fb15b75fac4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:48:13.382056 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.382036    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-home" (OuterVolumeSpecName: "home") pod "29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" (UID: "29043fbe-c2ca-48bc-a0a9-9fb15b75fac4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:48:13.382125 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.382075    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:48:13.384167 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.384127    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" (UID: "29043fbe-c2ca-48bc-a0a9-9fb15b75fac4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:48:13.384318 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.384295    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-dshm" (OuterVolumeSpecName: "dshm") pod "29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" (UID: "29043fbe-c2ca-48bc-a0a9-9fb15b75fac4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:48:13.384979 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.384959    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-kube-api-access-ghj78" (OuterVolumeSpecName: "kube-api-access-ghj78") pod "29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" (UID: "29043fbe-c2ca-48bc-a0a9-9fb15b75fac4"). InnerVolumeSpecName "kube-api-access-ghj78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:48:13.386627 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.386607    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0" (OuterVolumeSpecName: "kserve-pvc-source") pod "29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" (UID: "29043fbe-c2ca-48bc-a0a9-9fb15b75fac4"). InnerVolumeSpecName "pvc-bada7467-ebdd-48ed-8525-cac06e0bda69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 08 07:48:13.394332 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.394306    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" (UID: "29043fbe-c2ca-48bc-a0a9-9fb15b75fac4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:48:13.482554 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.482535    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:48:13.482846 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.482557    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:48:13.482846 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.482572    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:48:13.482846 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.482582    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghj78\" (UniqueName: \"kubernetes.io/projected/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-kube-api-access-ghj78\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:48:13.482846 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.482610    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") on node \"ip-10-0-140-110.ec2.internal\" "
Jul 08 07:48:13.482846 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.482619    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:48:13.511083 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.511068    2582 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-bada7467-ebdd-48ed-8525-cac06e0bda69" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0") on node "ip-10-0-140-110.ec2.internal"
Jul 08 07:48:13.583661 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.583632    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"csi-39614a8fa7302f6a5e404233f1c9304664ec83103aea2cf5242a35c67533acd5\""
Jul 08 07:48:13.850475 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.850447    2582 generic.go:356] "Generic (PLEG): container finished" podID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerID="d4f828506edbebce64d821529fb4236eb72089f646e8c95d1757d90871309010" exitCode=0
Jul 08 07:48:13.850613 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.850496    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"
Jul 08 07:48:13.850613 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.850536    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" event={"ID":"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4","Type":"ContainerDied","Data":"d4f828506edbebce64d821529fb4236eb72089f646e8c95d1757d90871309010"}
Jul 08 07:48:13.850613 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.850590    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" event={"ID":"29043fbe-c2ca-48bc-a0a9-9fb15b75fac4","Type":"ContainerDied","Data":"c9c99e758d0d78de4bb2567da5280f013b162989dfad5a122589585b17ed3e7d"}
Jul 08 07:48:13.850729 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.850614    2582 scope.go:117] "RemoveContainer" containerID="d4f828506edbebce64d821529fb4236eb72089f646e8c95d1757d90871309010"
Jul 08 07:48:13.860850 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.860833    2582 scope.go:117] "RemoveContainer" containerID="d4f828506edbebce64d821529fb4236eb72089f646e8c95d1757d90871309010"
Jul 08 07:48:13.861091 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:48:13.861074    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f828506edbebce64d821529fb4236eb72089f646e8c95d1757d90871309010\": container with ID starting with d4f828506edbebce64d821529fb4236eb72089f646e8c95d1757d90871309010 not found: ID does not exist" containerID="d4f828506edbebce64d821529fb4236eb72089f646e8c95d1757d90871309010"
Jul 08 07:48:13.861191 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.861097    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f828506edbebce64d821529fb4236eb72089f646e8c95d1757d90871309010"} err="failed to get container status \"d4f828506edbebce64d821529fb4236eb72089f646e8c95d1757d90871309010\": rpc error: code = NotFound desc = could not find container \"d4f828506edbebce64d821529fb4236eb72089f646e8c95d1757d90871309010\": container with ID starting with d4f828506edbebce64d821529fb4236eb72089f646e8c95d1757d90871309010 not found: ID does not exist"
Jul 08 07:48:13.873171 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.873130    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"]
Jul 08 07:48:13.880797 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:13.880773    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j"]
Jul 08 07:48:14.222616 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:14.222575    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5dz4j" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 08 07:48:14.745567 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:14.745524    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" path="/var/lib/kubelet/pods/29043fbe-c2ca-48bc-a0a9-9fb15b75fac4/volumes"
Jul 08 07:48:20.887417 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:20.887325    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817" exitCode=137
Jul 08 07:48:20.887417 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:20.887361    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerDied","Data":"dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"}
Jul 08 07:48:20.887917 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:20.887408    2582 scope.go:117] "RemoveContainer" containerID="1661058b3d1bf0b4c5f6fc08c4c5a05b2f4ec19488679b0c52c6225430a58a6c"
Jul 08 07:48:20.887917 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:20.887750    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:48:20.888019 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:48:20.887949    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:48:30.656891 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:30.656854    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:48:30.657370 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:30.657291    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:48:30.657497 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:48:30.657476    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:48:40.540348 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.540265    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"]
Jul 08 07:48:40.540746 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.540613    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main"
Jul 08 07:48:40.540746 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.540624    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main"
Jul 08 07:48:40.540746 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.540690    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="29043fbe-c2ca-48bc-a0a9-9fb15b75fac4" containerName="main"
Jul 08 07:48:40.543675 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.543657    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.547144 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.547121    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-76tcv\""
Jul 08 07:48:40.547594 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.547571    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 08 07:48:40.574670 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.574635    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"]
Jul 08 07:48:40.576373 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.576346    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"]
Jul 08 07:48:40.580213 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.580192    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.602924 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.602889    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"]
Jul 08 07:48:40.627557 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.627520    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.627720 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.627607    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.627720 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.627669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf33de-bca0-41d8-84c9-bd15b833a672-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.627720 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.627694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.627886 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.627738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.630596 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.630558    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:48:40.728780 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.728736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.728993 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.728807    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.728993 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.728847    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60d5800d-8fba-4001-851e-5d45f085e4fc-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.728993 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.728875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.728993 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.728897    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.728993 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.728920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p69q8\" (UniqueName: \"kubernetes.io/projected/60d5800d-8fba-4001-851e-5d45f085e4fc-kube-api-access-p69q8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.728993 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.728947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.729364 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.729046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf33de-bca0-41d8-84c9-bd15b833a672-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.729364 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.729084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.729364 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.729132    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdhvn\" (UniqueName: \"kubernetes.io/projected/6faf33de-bca0-41d8-84c9-bd15b833a672-kube-api-access-gdhvn\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.729364 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.729208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.729364 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.729254    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.729364 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.729334    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.729682 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.729487    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.729682 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.729647    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.731410 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.731375    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.731722 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.731695    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf33de-bca0-41d8-84c9-bd15b833a672-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.830249 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.830127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhvn\" (UniqueName: \"kubernetes.io/projected/6faf33de-bca0-41d8-84c9-bd15b833a672-kube-api-access-gdhvn\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.830423 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.830286    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.830423 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.830372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.830423 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.830404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60d5800d-8fba-4001-851e-5d45f085e4fc-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.830617 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.830455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.830617 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.830486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p69q8\" (UniqueName: \"kubernetes.io/projected/60d5800d-8fba-4001-851e-5d45f085e4fc-kube-api-access-p69q8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.830617 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.830537    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.830776 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.830725    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.830842 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.830770    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.831536 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.831512    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.833389 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.833363    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.833763 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.833743    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60d5800d-8fba-4001-851e-5d45f085e4fc-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:40.840206 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.840180    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhvn\" (UniqueName: \"kubernetes.io/projected/6faf33de-bca0-41d8-84c9-bd15b833a672-kube-api-access-gdhvn\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:40.840498 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:40.840477    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p69q8\" (UniqueName: \"kubernetes.io/projected/60d5800d-8fba-4001-851e-5d45f085e4fc-kube-api-access-p69q8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:41.040419 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:41.040382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:48:43.741440 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:43.741405    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:48:43.741995 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:48:43.741666    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:48:49.717686 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:49.717649    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:49.729357 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:49.729326    2582 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:49.819102 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:49.819065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:49.819272 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:49.819219    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:49.822879 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:49.822855    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") DevicePath \"csi-39614a8fa7302f6a5e404233f1c9304664ec83103aea2cf5242a35c67533acd5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:49.916381 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:49.916347    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d8cc943a74105dc3dfe7dcd9ae9ba70ab2d5536dabcf5b22935bc499f06ab79b/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:49.935071 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:49.935037    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:50.020603 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:50.020521    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:50.020734 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:50.020683    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") DevicePath \"csi-39614a8fa7302f6a5e404233f1c9304664ec83103aea2cf5242a35c67533acd5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:50.024725 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:50.024701    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") DevicePath \"csi-39614a8fa7302f6a5e404233f1c9304664ec83103aea2cf5242a35c67533acd5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:50.036679 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:50.036656    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:50.158520 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:50.158494    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-76tcv\""
Jul 08 07:48:50.164177 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:50.164132    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:50.190031 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:50.190006    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:48:50.323584 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:50.323546    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"]
Jul 08 07:48:50.327267 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:48:50.327236    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6faf33de_bca0_41d8_84c9_bd15b833a672.slice/crio-5b34bdc792ca08af3fcfdfe217c9fbbe7941cf11b959b1ce4915f67eeeb4c76c WatchSource:0}: Error finding container 5b34bdc792ca08af3fcfdfe217c9fbbe7941cf11b959b1ce4915f67eeeb4c76c: Status 404 returned error can't find the container with id 5b34bdc792ca08af3fcfdfe217c9fbbe7941cf11b959b1ce4915f67eeeb4c76c
Jul 08 07:48:50.330002 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:50.329983    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 07:48:50.348630 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:50.348608    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"]
Jul 08 07:48:50.350014 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:48:50.349992    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d5800d_8fba_4001_851e_5d45f085e4fc.slice/crio-2a18c4e1822e6fd3d4802314955abdc9ee95b6e83a5f269864251f8fd402076f WatchSource:0}: Error finding container 2a18c4e1822e6fd3d4802314955abdc9ee95b6e83a5f269864251f8fd402076f: Status 404 returned error can't find the container with id 2a18c4e1822e6fd3d4802314955abdc9ee95b6e83a5f269864251f8fd402076f
Jul 08 07:48:51.041173 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:51.041110    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" event={"ID":"60d5800d-8fba-4001-851e-5d45f085e4fc","Type":"ContainerStarted","Data":"4cf075e875a66b3e33bf1f6fa6e89bfef7daf95b2e6a0140fb6bad4086ed1405"}
Jul 08 07:48:51.041672 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:51.041212    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" event={"ID":"60d5800d-8fba-4001-851e-5d45f085e4fc","Type":"ContainerStarted","Data":"2a18c4e1822e6fd3d4802314955abdc9ee95b6e83a5f269864251f8fd402076f"}
Jul 08 07:48:51.042612 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:51.042584    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" event={"ID":"6faf33de-bca0-41d8-84c9-bd15b833a672","Type":"ContainerStarted","Data":"5b34bdc792ca08af3fcfdfe217c9fbbe7941cf11b959b1ce4915f67eeeb4c76c"}
Jul 08 07:48:51.064404 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:51.064342    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podStartSLOduration=11.064324934 podStartE2EDuration="11.064324934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 07:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:48:51.060837941 +0000 UTC m=+3246.864624365" watchObservedRunningTime="2026-07-08 07:48:51.064324934 +0000 UTC m=+3246.868111314"
Jul 08 07:48:52.050321 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:52.050282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" event={"ID":"6faf33de-bca0-41d8-84c9-bd15b833a672","Type":"ContainerStarted","Data":"66576862dbc892bb95e61c4d289f26aced3e52432679e19d5534c62bce410e86"}
Jul 08 07:48:52.050838 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:52.050507    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:48:53.058579 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:53.058537    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" event={"ID":"6faf33de-bca0-41d8-84c9-bd15b833a672","Type":"ContainerStarted","Data":"dc874aae2a0c30288e73255d7ea90e6eb0259e092731e16de81013d0443759f2"}
Jul 08 07:48:53.084215 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:53.084135    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podStartSLOduration=12.18540579 podStartE2EDuration="13.084114117s" totalImagesPullingTime="898.708327ms" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 07:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:48:53.081894203 +0000 UTC m=+3248.885680607" watchObservedRunningTime="2026-07-08 07:48:53.084114117 +0000 UTC m=+3248.887900521"
Jul 08 07:48:55.390783 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:55.390746    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:48:58.741698 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:48:58.741668    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:48:58.742188 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:48:58.741898    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:49:00.164717 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:00.164678    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:49:00.165188 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:00.164749    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:49:00.166201 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:00.166167    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:49:00.190874 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:00.190835    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:49:00.191062 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:00.190896    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:49:00.192438 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:00.192405    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:49:08.263283 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:08.263245    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:49:10.165083 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:10.165022    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:49:10.182432 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:10.182401    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:49:10.190432 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:10.190393    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:49:13.740984 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:13.740949    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:49:13.741613 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:49:13.741146    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:49:20.164918 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:20.164864    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:49:20.190816 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:20.190784    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:49:24.743364 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:24.743324    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:49:24.743863 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:49:24.743538    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:49:30.164860 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:30.164816    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:49:30.191527 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:30.191483    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:49:38.741376 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:38.741337    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:49:38.741763 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:49:38.741630    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:49:40.165407 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:40.165357    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:49:40.191092 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:40.191047    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:49:44.911495 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:44.911464    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:49:44.918908 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:44.918885    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:49:50.165041 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:50.164990    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:49:50.191258 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:50.191203    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:49:50.741000 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:49:50.740967    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:49:50.741243 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:49:50.741198    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:50:00.165249 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:00.165192    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:50:00.190796 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:00.190757    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:50:03.740727 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:03.740685    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:50:03.741291 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:50:03.740955    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:50:10.164761 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:10.164647    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:50:10.191220 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:10.191171    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:50:14.744455 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:14.744425    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:50:14.744839 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:50:14.744619    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:50:20.165354 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:20.165311    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:50:20.191385 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:20.191345    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:50:26.741295 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:26.741258    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:50:26.741798 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:50:26.741449    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:50:30.165113 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:30.165061    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:50:30.190488 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:30.190446    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:50:39.740839 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:39.740793    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:50:39.741394 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:50:39.740975    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:50:40.164872 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:40.164826    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:50:40.190433 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:40.190390    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:50:50.164769 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:50.164719    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:50:50.190877 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:50.190835    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:50:51.741532 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:51.741497    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:50:51.742008 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:50:51.741748    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:50:52.701706 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:52.701657    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:50:53.726062 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:50:53.726022    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:51:00.165455 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:00.165404    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:51:00.190600 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:00.190551    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:51:06.740797 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:06.740760    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:51:06.741356 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:51:06.740951    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:51:10.164984 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:10.164933    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:51:10.191014 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:10.190975    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:51:17.741112 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:17.741081    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:51:17.741533 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:51:17.741300    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:51:20.165294 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:20.165232    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:51:20.191328 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:20.191281    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:51:27.905766 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:27.905724    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:51:30.165106 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:30.165056    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:51:30.191167 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:30.191099    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:51:32.741455 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:32.741421    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:51:32.741844 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:51:32.741663    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:51:40.165656 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:40.165546    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 08 07:51:40.190643 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:40.190599    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 08 07:51:47.740913 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:47.740875    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:51:47.741422 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:51:47.741074    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:51:50.174303 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:50.174272    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:51:50.186815 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:50.186788    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:51:50.201261 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:50.201222    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:51:50.215427 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:51:50.215395    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:52:00.740956 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:52:00.740923    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:52:00.741372 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:52:00.741124    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:52:14.742791 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:52:14.742750    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:52:14.743307 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:52:14.742940    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:52:25.740578 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:52:25.740542    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:52:25.741072 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:52:25.740732    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:52:38.741448 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:52:38.741414    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:52:38.741945 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:52:38.741570    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:52:50.740706 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:52:50.740671    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:52:50.741093 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:52:50.740918    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:53:03.741003 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:03.740970    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:53:03.741538 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:53:03.741183    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:53:16.741274 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:16.741197    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:53:16.741633 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:53:16.741483    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:53:28.741233 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:28.741202    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:53:28.742543 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:28.742505    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:53:28.742627 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:28.742605    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:53:28.742670 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:28.742656    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:53:29.491494 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:29.491451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerStarted","Data":"ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"}
Jul 08 07:53:29.515209 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:29.515179    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:53:29.516333 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:29.516307    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:53:29.516459 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:29.516357    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:53:30.514849 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:30.514818    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:53:30.515267 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:30.514873    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:53:30.656386 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:30.656350    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:53:31.515705 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:31.515659    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:53:31.516126 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:31.515711    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:53:32.515248 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:32.515213    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:53:32.515445 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:32.515272    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:53:33.516751 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:33.516705    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:53:33.517271 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:33.516770    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:53:34.515503 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:34.515464    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:53:34.515714 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:34.515535    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:53:36.516276 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:36.516229    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 07:53:36.516934 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:36.516299    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 07:53:37.558340 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:37.558301    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 07:53:37.559767 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:37.558365    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 07:53:37.665480 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:37.665449    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:53:37.666446 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:53:37.666422    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:54:00.708863 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:00.708824    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"]
Jul 08 07:54:00.719076 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:00.719046    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"]
Jul 08 07:54:00.808552 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:00.808521    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:54:00.945383 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:00.945344    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:54:15.737178 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:15.737124    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main" containerID="cri-o://4cf075e875a66b3e33bf1f6fa6e89bfef7daf95b2e6a0140fb6bad4086ed1405" gracePeriod=45
Jul 08 07:54:15.742289 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:15.742234    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main" containerID="cri-o://dc874aae2a0c30288e73255d7ea90e6eb0259e092731e16de81013d0443759f2" gracePeriod=45
Jul 08 07:54:16.665347 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.665281    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="llm-d-routing-sidecar" containerID="cri-o://66576862dbc892bb95e61c4d289f26aced3e52432679e19d5534c62bce410e86" gracePeriod=45
Jul 08 07:54:16.759046 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.759008    2582 generic.go:356] "Generic (PLEG): container finished" podID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerID="4cf075e875a66b3e33bf1f6fa6e89bfef7daf95b2e6a0140fb6bad4086ed1405" exitCode=0
Jul 08 07:54:16.759532 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.759093    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" event={"ID":"60d5800d-8fba-4001-851e-5d45f085e4fc","Type":"ContainerDied","Data":"4cf075e875a66b3e33bf1f6fa6e89bfef7daf95b2e6a0140fb6bad4086ed1405"}
Jul 08 07:54:16.761559 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.761533    2582 generic.go:356] "Generic (PLEG): container finished" podID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerID="dc874aae2a0c30288e73255d7ea90e6eb0259e092731e16de81013d0443759f2" exitCode=0
Jul 08 07:54:16.761688 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.761570    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" event={"ID":"6faf33de-bca0-41d8-84c9-bd15b833a672","Type":"ContainerDied","Data":"dc874aae2a0c30288e73255d7ea90e6eb0259e092731e16de81013d0443759f2"}
Jul 08 07:54:16.844821 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.844743    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:54:16.962086 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.962060    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:54:16.970434 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.970405    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-model-cache\") pod \"60d5800d-8fba-4001-851e-5d45f085e4fc\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") "
Jul 08 07:54:16.970597 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.970520    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-home\") pod \"60d5800d-8fba-4001-851e-5d45f085e4fc\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") "
Jul 08 07:54:16.970597 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.970552    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-dshm\") pod \"60d5800d-8fba-4001-851e-5d45f085e4fc\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") "
Jul 08 07:54:16.970597 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.970587    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p69q8\" (UniqueName: \"kubernetes.io/projected/60d5800d-8fba-4001-851e-5d45f085e4fc-kube-api-access-p69q8\") pod \"60d5800d-8fba-4001-851e-5d45f085e4fc\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") "
Jul 08 07:54:16.970773 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.970624    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-tmp-dir\") pod \"60d5800d-8fba-4001-851e-5d45f085e4fc\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") "
Jul 08 07:54:16.970773 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.970701    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "60d5800d-8fba-4001-851e-5d45f085e4fc" (UID: "60d5800d-8fba-4001-851e-5d45f085e4fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:54:16.970773 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.970737    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"60d5800d-8fba-4001-851e-5d45f085e4fc\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") "
Jul 08 07:54:16.970934 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.970788    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60d5800d-8fba-4001-851e-5d45f085e4fc-tls-certs\") pod \"60d5800d-8fba-4001-851e-5d45f085e4fc\" (UID: \"60d5800d-8fba-4001-851e-5d45f085e4fc\") "
Jul 08 07:54:16.971394 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.971083    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:54:16.971394 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.971316    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-home" (OuterVolumeSpecName: "home") pod "60d5800d-8fba-4001-851e-5d45f085e4fc" (UID: "60d5800d-8fba-4001-851e-5d45f085e4fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:54:16.976698 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.976655    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d5800d-8fba-4001-851e-5d45f085e4fc-kube-api-access-p69q8" (OuterVolumeSpecName: "kube-api-access-p69q8") pod "60d5800d-8fba-4001-851e-5d45f085e4fc" (UID: "60d5800d-8fba-4001-851e-5d45f085e4fc"). InnerVolumeSpecName "kube-api-access-p69q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:54:16.977743 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.977721    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0" (OuterVolumeSpecName: "kserve-pvc-source") pod "60d5800d-8fba-4001-851e-5d45f085e4fc" (UID: "60d5800d-8fba-4001-851e-5d45f085e4fc"). InnerVolumeSpecName "pvc-bada7467-ebdd-48ed-8525-cac06e0bda69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 08 07:54:16.977977 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.977940    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d5800d-8fba-4001-851e-5d45f085e4fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60d5800d-8fba-4001-851e-5d45f085e4fc" (UID: "60d5800d-8fba-4001-851e-5d45f085e4fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:54:16.984175 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.984128    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-dshm" (OuterVolumeSpecName: "dshm") pod "60d5800d-8fba-4001-851e-5d45f085e4fc" (UID: "60d5800d-8fba-4001-851e-5d45f085e4fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:54:16.989375 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:16.989338    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60d5800d-8fba-4001-851e-5d45f085e4fc" (UID: "60d5800d-8fba-4001-851e-5d45f085e4fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:54:17.071816 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.071780    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"6faf33de-bca0-41d8-84c9-bd15b833a672\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") "
Jul 08 07:54:17.072069 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.071839    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-model-cache\") pod \"6faf33de-bca0-41d8-84c9-bd15b833a672\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") "
Jul 08 07:54:17.072069 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.071885    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdhvn\" (UniqueName: \"kubernetes.io/projected/6faf33de-bca0-41d8-84c9-bd15b833a672-kube-api-access-gdhvn\") pod \"6faf33de-bca0-41d8-84c9-bd15b833a672\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") "
Jul 08 07:54:17.072069 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.071941    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-tmp-dir\") pod \"6faf33de-bca0-41d8-84c9-bd15b833a672\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") "
Jul 08 07:54:17.072069 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.072003    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-dshm\") pod \"6faf33de-bca0-41d8-84c9-bd15b833a672\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") "
Jul 08 07:54:17.072339 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.072093    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-model-cache" (OuterVolumeSpecName: "model-cache") pod "6faf33de-bca0-41d8-84c9-bd15b833a672" (UID: "6faf33de-bca0-41d8-84c9-bd15b833a672"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:54:17.072339 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.072133    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf33de-bca0-41d8-84c9-bd15b833a672-tls-certs\") pod \"6faf33de-bca0-41d8-84c9-bd15b833a672\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") "
Jul 08 07:54:17.072339 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.072196    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-home\") pod \"6faf33de-bca0-41d8-84c9-bd15b833a672\" (UID: \"6faf33de-bca0-41d8-84c9-bd15b833a672\") "
Jul 08 07:54:17.072582 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.072555    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:54:17.072661 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.072589    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:54:17.072661 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.072604    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60d5800d-8fba-4001-851e-5d45f085e4fc-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:54:17.072661 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.072618    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:54:17.072661 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.072632    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60d5800d-8fba-4001-851e-5d45f085e4fc-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:54:17.072661 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.072646    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p69q8\" (UniqueName: \"kubernetes.io/projected/60d5800d-8fba-4001-851e-5d45f085e4fc-kube-api-access-p69q8\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:54:17.072964 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.072938    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-home" (OuterVolumeSpecName: "home") pod "6faf33de-bca0-41d8-84c9-bd15b833a672" (UID: "6faf33de-bca0-41d8-84c9-bd15b833a672"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:54:17.074370 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.074345    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-dshm" (OuterVolumeSpecName: "dshm") pod "6faf33de-bca0-41d8-84c9-bd15b833a672" (UID: "6faf33de-bca0-41d8-84c9-bd15b833a672"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:54:17.074726 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.074703    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6faf33de-bca0-41d8-84c9-bd15b833a672-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6faf33de-bca0-41d8-84c9-bd15b833a672" (UID: "6faf33de-bca0-41d8-84c9-bd15b833a672"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:54:17.074845 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.074829    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6faf33de-bca0-41d8-84c9-bd15b833a672-kube-api-access-gdhvn" (OuterVolumeSpecName: "kube-api-access-gdhvn") pod "6faf33de-bca0-41d8-84c9-bd15b833a672" (UID: "6faf33de-bca0-41d8-84c9-bd15b833a672"). InnerVolumeSpecName "kube-api-access-gdhvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:54:17.074979 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.074960    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0" (OuterVolumeSpecName: "kserve-pvc-source") pod "6faf33de-bca0-41d8-84c9-bd15b833a672" (UID: "6faf33de-bca0-41d8-84c9-bd15b833a672"). InnerVolumeSpecName "pvc-bada7467-ebdd-48ed-8525-cac06e0bda69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 08 07:54:17.087103 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.087076    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6faf33de-bca0-41d8-84c9-bd15b833a672" (UID: "6faf33de-bca0-41d8-84c9-bd15b833a672"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:54:17.174062 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.174030    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:54:17.174062 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.174061    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6faf33de-bca0-41d8-84c9-bd15b833a672-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:54:17.174324 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.174072    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:54:17.174324 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.174104    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") on node \"ip-10-0-140-110.ec2.internal\" "
Jul 08 07:54:17.174324 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.174117    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdhvn\" (UniqueName: \"kubernetes.io/projected/6faf33de-bca0-41d8-84c9-bd15b833a672-kube-api-access-gdhvn\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:54:17.174324 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.174126    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6faf33de-bca0-41d8-84c9-bd15b833a672-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 07:54:17.201869 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.201843    2582 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-bada7467-ebdd-48ed-8525-cac06e0bda69" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0") on node "ip-10-0-140-110.ec2.internal"
Jul 08 07:54:17.275059 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.275020    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"csi-39614a8fa7302f6a5e404233f1c9304664ec83103aea2cf5242a35c67533acd5\""
Jul 08 07:54:17.768590 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.768562    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95" exitCode=137
Jul 08 07:54:17.768999 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.768638    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerDied","Data":"ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"}
Jul 08 07:54:17.768999 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.768717    2582 scope.go:117] "RemoveContainer" containerID="dc8908f7dba27980a1d4be74745fd9811a9012544b0b403b62eb8a1d7026b817"
Jul 08 07:54:17.769207 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.769192    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:54:17.769440 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:54:17.769415    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:54:17.770079 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.770056    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"
Jul 08 07:54:17.770196 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.770101    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q" event={"ID":"60d5800d-8fba-4001-851e-5d45f085e4fc","Type":"ContainerDied","Data":"2a18c4e1822e6fd3d4802314955abdc9ee95b6e83a5f269864251f8fd402076f"}
Jul 08 07:54:17.771657 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.771636    2582 generic.go:356] "Generic (PLEG): container finished" podID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerID="66576862dbc892bb95e61c4d289f26aced3e52432679e19d5534c62bce410e86" exitCode=0
Jul 08 07:54:17.771746 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.771689    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" event={"ID":"6faf33de-bca0-41d8-84c9-bd15b833a672","Type":"ContainerDied","Data":"66576862dbc892bb95e61c4d289f26aced3e52432679e19d5534c62bce410e86"}
Jul 08 07:54:17.771746 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.771710    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"
Jul 08 07:54:17.771746 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.771724    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4" event={"ID":"6faf33de-bca0-41d8-84c9-bd15b833a672","Type":"ContainerDied","Data":"5b34bdc792ca08af3fcfdfe217c9fbbe7941cf11b959b1ce4915f67eeeb4c76c"}
Jul 08 07:54:17.790288 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.790105    2582 scope.go:117] "RemoveContainer" containerID="4cf075e875a66b3e33bf1f6fa6e89bfef7daf95b2e6a0140fb6bad4086ed1405"
Jul 08 07:54:17.797979 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.797960    2582 scope.go:117] "RemoveContainer" containerID="dc874aae2a0c30288e73255d7ea90e6eb0259e092731e16de81013d0443759f2"
Jul 08 07:54:17.805459 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.805438    2582 scope.go:117] "RemoveContainer" containerID="66576862dbc892bb95e61c4d289f26aced3e52432679e19d5534c62bce410e86"
Jul 08 07:54:17.813256 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.813237    2582 scope.go:117] "RemoveContainer" containerID="dc874aae2a0c30288e73255d7ea90e6eb0259e092731e16de81013d0443759f2"
Jul 08 07:54:17.813494 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:54:17.813476    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc874aae2a0c30288e73255d7ea90e6eb0259e092731e16de81013d0443759f2\": container with ID starting with dc874aae2a0c30288e73255d7ea90e6eb0259e092731e16de81013d0443759f2 not found: ID does not exist" containerID="dc874aae2a0c30288e73255d7ea90e6eb0259e092731e16de81013d0443759f2"
Jul 08 07:54:17.813554 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.813502    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc874aae2a0c30288e73255d7ea90e6eb0259e092731e16de81013d0443759f2"} err="failed to get container status \"dc874aae2a0c30288e73255d7ea90e6eb0259e092731e16de81013d0443759f2\": rpc error: code = NotFound desc = could not find container \"dc874aae2a0c30288e73255d7ea90e6eb0259e092731e16de81013d0443759f2\": container with ID starting with dc874aae2a0c30288e73255d7ea90e6eb0259e092731e16de81013d0443759f2 not found: ID does not exist"
Jul 08 07:54:17.813554 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.813525    2582 scope.go:117] "RemoveContainer" containerID="66576862dbc892bb95e61c4d289f26aced3e52432679e19d5534c62bce410e86"
Jul 08 07:54:17.813754 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:54:17.813738    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66576862dbc892bb95e61c4d289f26aced3e52432679e19d5534c62bce410e86\": container with ID starting with 66576862dbc892bb95e61c4d289f26aced3e52432679e19d5534c62bce410e86 not found: ID does not exist" containerID="66576862dbc892bb95e61c4d289f26aced3e52432679e19d5534c62bce410e86"
Jul 08 07:54:17.813795 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.813760    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66576862dbc892bb95e61c4d289f26aced3e52432679e19d5534c62bce410e86"} err="failed to get container status \"66576862dbc892bb95e61c4d289f26aced3e52432679e19d5534c62bce410e86\": rpc error: code = NotFound desc = could not find container \"66576862dbc892bb95e61c4d289f26aced3e52432679e19d5534c62bce410e86\": container with ID starting with 66576862dbc892bb95e61c4d289f26aced3e52432679e19d5534c62bce410e86 not found: ID does not exist"
Jul 08 07:54:17.842410 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.842381    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"]
Jul 08 07:54:17.858697 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.858661    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-9k9f4"]
Jul 08 07:54:17.886058 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.886027    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"]
Jul 08 07:54:17.902206 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:17.902176    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79wcg6q"]
Jul 08 07:54:18.746458 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:18.746426    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" path="/var/lib/kubelet/pods/60d5800d-8fba-4001-851e-5d45f085e4fc/volumes"
Jul 08 07:54:18.746850 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:18.746838    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" path="/var/lib/kubelet/pods/6faf33de-bca0-41d8-84c9-bd15b833a672/volumes"
Jul 08 07:54:22.666448 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:22.666415    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:54:22.666924 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:22.666907    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:54:22.667124 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:54:22.667104    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:54:26.067703 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.067667    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:54:26.200249 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.200219    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:54:26.366075 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.365985    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:54:26.410949 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.410912    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 08 07:54:26.411332 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.411316    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main"
Jul 08 07:54:26.411378 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.411334    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main"
Jul 08 07:54:26.411378 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.411353    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="llm-d-routing-sidecar"
Jul 08 07:54:26.411378 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.411362    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="llm-d-routing-sidecar"
Jul 08 07:54:26.411378 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.411372    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main"
Jul 08 07:54:26.411378 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.411377    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main"
Jul 08 07:54:26.411540 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.411429    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="llm-d-routing-sidecar"
Jul 08 07:54:26.411540 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.411441    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="60d5800d-8fba-4001-851e-5d45f085e4fc" containerName="main"
Jul 08 07:54:26.411540 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.411449    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6faf33de-bca0-41d8-84c9-bd15b833a672" containerName="main"
Jul 08 07:54:26.415691 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.415673    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.419744 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.419718    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 08 07:54:26.419744 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.419738    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-zg5nt\""
Jul 08 07:54:26.440744 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.440714    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 08 07:54:26.477337 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.477290    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 08 07:54:26.480865 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.480838    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.508748 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.508718    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 08 07:54:26.559928 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.559890    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.560098 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.559954    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.560098 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.559976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d4f055-cfec-400b-ac7c-666b216ccaca-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.560098 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.560031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kqnx\" (UniqueName: \"kubernetes.io/projected/48d4f055-cfec-400b-ac7c-666b216ccaca-kube-api-access-4kqnx\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.560098 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.560052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.560098 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.560070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.560098 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.560094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.560373 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.560114    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48d4f055-cfec-400b-ac7c-666b216ccaca-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.560373 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.560147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48d4f055-cfec-400b-ac7c-666b216ccaca-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.560373 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.560198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48d4f055-cfec-400b-ac7c-666b216ccaca-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.560373 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.560225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgrz9\" (UniqueName: \"kubernetes.io/projected/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-kube-api-access-cgrz9\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.560373 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.560262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48d4f055-cfec-400b-ac7c-666b216ccaca-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.620016 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.619937    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:54:26.661804 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.661766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d4f055-cfec-400b-ac7c-666b216ccaca-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.662020 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.661820    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kqnx\" (UniqueName: \"kubernetes.io/projected/48d4f055-cfec-400b-ac7c-666b216ccaca-kube-api-access-4kqnx\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.662020 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.661987    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.662146 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.662146 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.662146 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48d4f055-cfec-400b-ac7c-666b216ccaca-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.662146 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662097    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48d4f055-cfec-400b-ac7c-666b216ccaca-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.662146 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48d4f055-cfec-400b-ac7c-666b216ccaca-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.662400 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662191    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d4f055-cfec-400b-ac7c-666b216ccaca-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.662400 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgrz9\" (UniqueName: \"kubernetes.io/projected/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-kube-api-access-cgrz9\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.662400 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662241    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.662400 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48d4f055-cfec-400b-ac7c-666b216ccaca-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.662400 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.662646 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662394    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.662646 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.662646 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662491    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48d4f055-cfec-400b-ac7c-666b216ccaca-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.662785 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662708    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.662785 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662750    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.662906 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.662884    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48d4f055-cfec-400b-ac7c-666b216ccaca-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.664694 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.664670    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48d4f055-cfec-400b-ac7c-666b216ccaca-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.664878 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.664862    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.664949 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.664930    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48d4f055-cfec-400b-ac7c-666b216ccaca-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.665030 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.665008    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.682027 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.682000    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kqnx\" (UniqueName: \"kubernetes.io/projected/48d4f055-cfec-400b-ac7c-666b216ccaca-kube-api-access-4kqnx\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.682649 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.682627    2582 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.685481 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.685463    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgrz9\" (UniqueName: \"kubernetes.io/projected/9d37b6b1-ed42-4a7e-aa20-371c401b8c99-kube-api-access-cgrz9\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.762942 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.762909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.763130 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.763064    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.771398 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.771368    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") DevicePath \"csi-39614a8fa7302f6a5e404233f1c9304664ec83103aea2cf5242a35c67533acd5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.840313 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.840281    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d8cc943a74105dc3dfe7dcd9ae9ba70ab2d5536dabcf5b22935bc499f06ab79b/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.853012 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.852980    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"48d4f055-cfec-400b-ac7c-666b216ccaca\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:26.863843 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.863816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.863976 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.863923    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") DevicePath \"csi-39614a8fa7302f6a5e404233f1c9304664ec83103aea2cf5242a35c67533acd5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.867622 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.867599    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") DevicePath \"csi-39614a8fa7302f6a5e404233f1c9304664ec83103aea2cf5242a35c67533acd5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:26.883280 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:26.883209    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bada7467-ebdd-48ed-8525-cac06e0bda69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03a3b4847c5ea08e0\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"9d37b6b1-ed42-4a7e-aa20-371c401b8c99\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:27.026417 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:27.026376    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:27.091203 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:27.091174    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 07:54:27.177933 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:27.177890    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 08 07:54:27.179196 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:54:27.179130    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d37b6b1_ed42_4a7e_aa20_371c401b8c99.slice/crio-75a9531b462c20eaaa131b0aa19e4e7965df834d255a8622e646a5f0283dc6c2 WatchSource:0}: Error finding container 75a9531b462c20eaaa131b0aa19e4e7965df834d255a8622e646a5f0283dc6c2: Status 404 returned error can't find the container with id 75a9531b462c20eaaa131b0aa19e4e7965df834d255a8622e646a5f0283dc6c2
Jul 08 07:54:27.250749 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:27.250722    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 08 07:54:27.251807 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:54:27.251772    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d4f055_cfec_400b_ac7c_666b216ccaca.slice/crio-09bbd480fdfed4c706b20708ad02ebbb1e93e178efe6044539edcbed51ddb53e WatchSource:0}: Error finding container 09bbd480fdfed4c706b20708ad02ebbb1e93e178efe6044539edcbed51ddb53e: Status 404 returned error can't find the container with id 09bbd480fdfed4c706b20708ad02ebbb1e93e178efe6044539edcbed51ddb53e
Jul 08 07:54:27.829111 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:27.829060    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"48d4f055-cfec-400b-ac7c-666b216ccaca","Type":"ContainerStarted","Data":"4fefe27e8657e159f787fcd96351a3114d85a7758fe093ca33084d2e7658317d"}
Jul 08 07:54:27.829364 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:27.829137    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"48d4f055-cfec-400b-ac7c-666b216ccaca","Type":"ContainerStarted","Data":"09bbd480fdfed4c706b20708ad02ebbb1e93e178efe6044539edcbed51ddb53e"}
Jul 08 07:54:27.831580 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:27.831547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"9d37b6b1-ed42-4a7e-aa20-371c401b8c99","Type":"ContainerStarted","Data":"1351ed9f7de239c3fb33203448f37b552ad981cd820bd7d4feb0762b1d07a587"}
Jul 08 07:54:27.831743 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:27.831602    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"9d37b6b1-ed42-4a7e-aa20-371c401b8c99","Type":"ContainerStarted","Data":"75a9531b462c20eaaa131b0aa19e4e7965df834d255a8622e646a5f0283dc6c2"}
Jul 08 07:54:27.880269 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:27.880216    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=1.8802021309999999 podStartE2EDuration="1.880202131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 07:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:54:27.872048423 +0000 UTC m=+3583.675834823" watchObservedRunningTime="2026-07-08 07:54:27.880202131 +0000 UTC m=+3583.683988530"
Jul 08 07:54:28.027352 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:28.027314    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:35.741256 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:35.741214    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:54:35.741766 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:54:35.741429    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:54:37.027458 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:37.027415    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:54:37.029097 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:37.029062    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:54:41.286944 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:41.286888    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:54:44.948507 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:44.948452    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:54:44.960775 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:44.960749    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:54:47.027899 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:47.027841    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:54:47.741140 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:47.741109    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:54:47.741411 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:54:47.741387    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:54:50.561650 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:50.561608    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:54:57.026954 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:57.026882    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:54:59.740763 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:54:59.740722    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:54:59.741176 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:54:59.740994    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:55:07.027586 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:07.027526    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:55:14.742616 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:14.742581    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:55:14.743103 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:55:14.742777    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:55:17.027407 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:17.027344    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:55:26.740794 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:26.740762    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:55:26.741384 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:55:26.740952    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:55:27.027355 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:27.027230    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:55:37.027799 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:37.027741    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:55:38.673849 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:38.673810    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 07:55:38.695640 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:38.695607    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 07:55:39.741067 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:39.741035    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:55:39.741473 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:55:39.741245    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:55:47.027471 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:47.027401    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:55:53.740923 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:53.740888    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:55:53.741364 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:55:53.741180    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:55:57.027527 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:57.027469    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:55:58.660841 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.660784    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=92.660763948 podStartE2EDuration="1m32.660763948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 07:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:54:27.938761191 +0000 UTC m=+3583.742547595" watchObservedRunningTime="2026-07-08 07:55:58.660763948 +0000 UTC m=+3674.464550349"
Jul 08 07:55:58.661241 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.660931    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"]
Jul 08 07:55:58.664740 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.664720    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.670226 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.670203    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 08 07:55:58.696838 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.696808    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"]
Jul 08 07:55:58.778252 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.778213    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:55:58.826429 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.826388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-kserve-provision-location\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.826644 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.826440    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-home\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.826644 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.826468    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-dshm\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.826644 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.826530    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-model-cache\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.826644 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.826610    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49gkq\" (UniqueName: \"kubernetes.io/projected/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-kube-api-access-49gkq\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.826912 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.826685    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-tmp-dir\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.826912 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.826733    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-tls-certs\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.927619 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.927582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-tls-certs\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.927829 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.927666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-kserve-provision-location\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.927829 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.927699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-home\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.927829 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.927725    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-dshm\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.927829 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.927800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-model-cache\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.928071 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.927842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gkq\" (UniqueName: \"kubernetes.io/projected/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-kube-api-access-49gkq\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.928071 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.927910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-tmp-dir\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.928828 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.928775    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-kserve-provision-location\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.928970 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.928787    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-model-cache\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.929038 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.929021    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-tmp-dir\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.929315 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.929293    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-home\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.930643 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.930620    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-dshm\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.930741 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.930717    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-tls-certs\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.937854 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.937827    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gkq\" (UniqueName: \"kubernetes.io/projected/df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9-kube-api-access-49gkq\") pod \"router-with-refs-test-kserve-794b6f7745-fpslh\" (UID: \"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:58.978561 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:58.978522    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:55:59.159762 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:59.159715    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"]
Jul 08 07:55:59.167819 ip-10-0-140-110 kubenswrapper[2582]: W0708 07:55:59.167781    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3d2e0b_ce6b_4275_b4f5_dae9bf0960e9.slice/crio-f16b515e215aa9938b200d5c1eb0bdeb6e711dd7ec9852dcc3f14c348fe28579 WatchSource:0}: Error finding container f16b515e215aa9938b200d5c1eb0bdeb6e711dd7ec9852dcc3f14c348fe28579: Status 404 returned error can't find the container with id f16b515e215aa9938b200d5c1eb0bdeb6e711dd7ec9852dcc3f14c348fe28579
Jul 08 07:55:59.177716 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:59.177112    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 07:55:59.337822 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:59.337775    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" event={"ID":"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9","Type":"ContainerStarted","Data":"259e99435fc59eaf20dcf4f5c9f9167468b55ca52f5247c7371ebe618081f111"}
Jul 08 07:55:59.338027 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:55:59.337847    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" event={"ID":"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9","Type":"ContainerStarted","Data":"f16b515e215aa9938b200d5c1eb0bdeb6e711dd7ec9852dcc3f14c348fe28579"}
Jul 08 07:56:00.084984 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:00.084946    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:56:03.368196 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:03.368108    2582 generic.go:356] "Generic (PLEG): container finished" podID="df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9" containerID="259e99435fc59eaf20dcf4f5c9f9167468b55ca52f5247c7371ebe618081f111" exitCode=0
Jul 08 07:56:03.368196 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:03.368184    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" event={"ID":"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9","Type":"ContainerDied","Data":"259e99435fc59eaf20dcf4f5c9f9167468b55ca52f5247c7371ebe618081f111"}
Jul 08 07:56:04.376096 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:04.376055    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" event={"ID":"df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9","Type":"ContainerStarted","Data":"c47414320f5d5c56f7263389be5bcf9e8d407d0d3b9f0551649b450fd4fd98ac"}
Jul 08 07:56:04.411848 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:04.411783    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" podStartSLOduration=3.079769999 podStartE2EDuration="6.411761837s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.331991838s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:56:04.41159789 +0000 UTC m=+3680.215384291" watchObservedRunningTime="2026-07-08 07:56:04.411761837 +0000 UTC m=+3680.215548239"
Jul 08 07:56:04.979426 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:04.979388    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:56:05.741308 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:05.741270    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:56:05.741774 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:56:05.741522    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:56:07.027084 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:07.027023    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:56:08.979191 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:08.979142    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:56:08.980692 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:08.980665    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" podUID="df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 08 07:56:17.027616 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:17.027566    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:56:18.979461 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:18.979408    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" podUID="df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 08 07:56:19.741407 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:19.741375    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:56:19.741616 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:56:19.741596    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:56:27.027865 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:27.027810    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:56:28.979073 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:28.979029    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" podUID="df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 08 07:56:34.747659 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:34.747623    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:56:34.748048 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:56:34.747904    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:56:37.027065 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:37.026999    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:56:38.979692 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:38.979647    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" podUID="df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 08 07:56:45.740690 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:45.740655    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:56:45.741137 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:56:45.740919    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:56:47.027848 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:47.027782    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:56:48.979831 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:48.979761    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" podUID="df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 08 07:56:57.027650 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:57.027599    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="9d37b6b1-ed42-4a7e-aa20-371c401b8c99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 08 07:56:58.741040 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:58.741007    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:56:58.741457 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:56:58.741212    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:56:58.979988 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:56:58.979937    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" podUID="df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 08 07:57:07.039235 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:07.039203    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:57:07.046892 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:07.046865    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 07:57:08.979388 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:08.979336    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" podUID="df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 08 07:57:09.741339 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:09.741301    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:57:09.741591 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:57:09.741566    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:57:18.979387 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:18.979335    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" podUID="df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 08 07:57:21.741026 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:21.740996    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:57:21.741525 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:57:21.741176    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:57:22.086968 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:22.086881    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:57:22.188029 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:22.187992    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:57:28.978963 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:28.978914    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" podUID="df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 08 07:57:34.742504 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:34.742474    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:57:34.742889 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:57:34.742658    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:57:38.979423 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:38.979324    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" podUID="df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 08 07:57:48.741523 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:48.741485    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:57:48.742007 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:57:48.741677    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:57:48.978928 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:48.978883    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" podUID="df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 08 07:57:52.880798 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:52.880751    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:57:58.979461 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:57:58.979400    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh" podUID="df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 08 07:58:00.741202 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:58:00.741137    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:58:00.741604 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:58:00.741447    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:58:08.990325 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:58:08.990292    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:58:08.998303 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:58:08.998280    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-794b6f7745-fpslh"
Jul 08 07:58:11.881396 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:58:11.881357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 07:58:13.741233 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:58:13.741197    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:58:13.741612 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:58:13.741369    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:58:25.740746 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:58:25.740716    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:58:25.741238 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:58:25.740882    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:58:40.741174 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:58:40.741134    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:58:40.741656 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:58:40.741409    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:58:55.740615 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:58:55.740584    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:58:55.741057 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:58:55.740752    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:59:08.741572 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:08.741478    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:59:08.742140 ip-10-0-140-110 kubenswrapper[2582]: E0708 07:59:08.741759    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-9455cf8d5-g22cx_openshift-ingress(c1e1a16a-a06e-4b84-842c-6b4415e0f636)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636"
Jul 08 07:59:20.741022 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:20.740991    2582 scope.go:117] "RemoveContainer" containerID="ac32a05a5f4e4bd73c14858cc4a77020eb60976e896171c645b37bf0846b9a95"
Jul 08 07:59:20.742337 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:20.742298    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:59:20.742465 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:20.742374    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:59:20.742465 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:20.742407    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 08 07:59:21.424654 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:21.424623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" event={"ID":"c1e1a16a-a06e-4b84-842c-6b4415e0f636","Type":"ContainerStarted","Data":"e080d9e9223f91eceb749c39c333a9beccfd90140aa4365d6a18a787412691d8"}
Jul 08 07:59:21.514912 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:21.514870    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:59:21.515825 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:21.515787    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:59:21.515961 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:21.515845    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:59:22.515078 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:22.515037    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:59:22.515471 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:22.515106    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:59:22.666103 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:22.666054    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:59:23.514969 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:23.514938    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:59:23.515145 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:23.514987    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:59:24.515629 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:24.515592    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:59:24.515992 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:24.515650    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:59:25.515372 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:25.515337    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:59:25.515538 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:25.515415    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:59:26.515413 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:26.515378    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 08 07:59:26.515784 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:26.515435    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 08 07:59:28.529566 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:28.529522    2582 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-9455cf8d5-g22cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 07:59:28.530052 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:28.529840    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx" podUID="c1e1a16a-a06e-4b84-842c-6b4415e0f636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 07:59:29.364541 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:29.364508    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:59:29.365617 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:29.365593    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-9455cf8d5-g22cx"
Jul 08 07:59:44.987435 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:44.987404    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 07:59:44.997518 ip-10-0-140-110 kubenswrapper[2582]: I0708 07:59:44.997496    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:03:39.264658 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:03:39.264573    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:03:40.894344 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:03:40.894314    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:04:39.072068 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:04:39.072021    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:04:45.022937 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:04:45.022906    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:04:45.035581 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:04:45.035559    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:06:04.116928 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:06:04.116893    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:06:55.308908 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:06:55.308870    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:09:22.920310 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:22.920278    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:09:22.930599 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:22.930568    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:09:34.708632 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.708594    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"]
Jul 08 08:09:34.712274 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.712253    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.714832 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.714811    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 08 08:09:34.736592 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.736566    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"]
Jul 08 08:09:34.774542 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.766525    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kt5r\" (UniqueName: \"kubernetes.io/projected/72ec3281-e757-40fc-894c-131af6299600-kube-api-access-5kt5r\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.774542 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.766602    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-home\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.774542 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.766639    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-model-cache\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.774542 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.766812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ec3281-e757-40fc-894c-131af6299600-tls-certs\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.774542 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.766902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-tmp-dir\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.774542 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.766895    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:09:34.774542 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.766955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-dshm\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.774542 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.766991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-kserve-provision-location\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.868222 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.868191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-dshm\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.868222 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.868225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-kserve-provision-location\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.868490 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.868290    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kt5r\" (UniqueName: \"kubernetes.io/projected/72ec3281-e757-40fc-894c-131af6299600-kube-api-access-5kt5r\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.868490 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.868315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-home\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.868490 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.868330    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-model-cache\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.868490 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.868354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ec3281-e757-40fc-894c-131af6299600-tls-certs\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.868490 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.868435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-tmp-dir\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.868752 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.868729    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-home\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.868835 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.868807    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-kserve-provision-location\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.868835 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.868825    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-tmp-dir\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.869143 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.869121    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-model-cache\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.870844 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.870823    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-dshm\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.871548 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.871529    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ec3281-e757-40fc-894c-131af6299600-tls-certs\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:34.877009 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:34.876982    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kt5r\" (UniqueName: \"kubernetes.io/projected/72ec3281-e757-40fc-894c-131af6299600-kube-api-access-5kt5r\") pod \"stop-feature-test-kserve-7854d477c-n9w7p\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:35.022326 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:35.022250    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:35.183839 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:35.183803    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"]
Jul 08 08:09:35.185980 ip-10-0-140-110 kubenswrapper[2582]: W0708 08:09:35.185950    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ec3281_e757_40fc_894c_131af6299600.slice/crio-a445b096e884290fe8ccd2a9f6cc8eb7fce0829e6c8d87e67e987634a2980cbe WatchSource:0}: Error finding container a445b096e884290fe8ccd2a9f6cc8eb7fce0829e6c8d87e67e987634a2980cbe: Status 404 returned error can't find the container with id a445b096e884290fe8ccd2a9f6cc8eb7fce0829e6c8d87e67e987634a2980cbe
Jul 08 08:09:35.196467 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:35.196444    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 08:09:35.581310 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:35.581269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" event={"ID":"72ec3281-e757-40fc-894c-131af6299600","Type":"ContainerStarted","Data":"c53365d3781b92cd871a64f5666e0d3221b567905062a9f5096427bf016c6b65"}
Jul 08 08:09:35.581530 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:35.581339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" event={"ID":"72ec3281-e757-40fc-894c-131af6299600","Type":"ContainerStarted","Data":"a445b096e884290fe8ccd2a9f6cc8eb7fce0829e6c8d87e67e987634a2980cbe"}
Jul 08 08:09:36.466175 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:36.466116    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:09:41.623877 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:41.623841    2582 generic.go:356] "Generic (PLEG): container finished" podID="72ec3281-e757-40fc-894c-131af6299600" containerID="c53365d3781b92cd871a64f5666e0d3221b567905062a9f5096427bf016c6b65" exitCode=0
Jul 08 08:09:41.624356 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:41.623916    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" event={"ID":"72ec3281-e757-40fc-894c-131af6299600","Type":"ContainerDied","Data":"c53365d3781b92cd871a64f5666e0d3221b567905062a9f5096427bf016c6b65"}
Jul 08 08:09:42.636091 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:42.636050    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" event={"ID":"72ec3281-e757-40fc-894c-131af6299600","Type":"ContainerStarted","Data":"885073ef47e4a2079bc863e8de147388f84ab417e8c347374a78816325c4055a"}
Jul 08 08:09:42.664580 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:42.664523    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podStartSLOduration=3.222818137 podStartE2EDuration="8.66450135s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.441683213s" isStatefulPod=false podCreationTimestamp="2026-07-08 08:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 08:09:42.660568588 +0000 UTC m=+4498.464355014" watchObservedRunningTime="2026-07-08 08:09:42.66450135 +0000 UTC m=+4498.468287752"
Jul 08 08:09:43.023066 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:43.023023    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:45.023162 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:45.023073    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:09:45.024334 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:45.024307    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 08 08:09:45.055962 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:45.055932    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:09:45.070333 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:45.070309    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:09:49.213671 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:49.213632    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:09:55.023585 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:09:55.023528    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 08 08:10:05.023635 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:10:05.023580    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 08 08:10:15.023281 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:10:15.023232    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 08 08:10:25.023133 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:10:25.023077    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 08 08:10:35.022897 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:10:35.022847    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 08 08:10:45.022890 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:10:45.022792    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 08 08:10:45.585815 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:10:45.585778    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:10:45.594628 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:10:45.594603    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:10:55.023163 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:10:55.023106    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 08 08:11:05.023649 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:05.023604    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 08 08:11:13.403103 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.403009    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"]
Jul 08 08:11:13.406722 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.406698    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.409758 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.409737    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 08 08:11:13.471088 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.471061    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"]
Jul 08 08:11:13.486029 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.485978    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:11:13.512890 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.512861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.513038 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.512917    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2cc767-a049-4234-b165-4a0ce5f7bc37-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.513038 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.512980    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.513038 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.513027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.513260 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.513066    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.513260 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.513091    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st9ph\" (UniqueName: \"kubernetes.io/projected/bb2cc767-a049-4234-b165-4a0ce5f7bc37-kube-api-access-st9ph\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.513260 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.513111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.614784 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.614741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2cc767-a049-4234-b165-4a0ce5f7bc37-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.614970 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.614841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.614970 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.614879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.614970 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.614909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.614970 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.614949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st9ph\" (UniqueName: \"kubernetes.io/projected/bb2cc767-a049-4234-b165-4a0ce5f7bc37-kube-api-access-st9ph\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.615223 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.614981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.615223 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.615054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.615363 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.615307    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.615511 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.615486    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.615612 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.615597    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.615709 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.615687    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.617429 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.617409    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2cc767-a049-4234-b165-4a0ce5f7bc37-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.617537 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.617493    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.624234 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.624214    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st9ph\" (UniqueName: \"kubernetes.io/projected/bb2cc767-a049-4234-b165-4a0ce5f7bc37-kube-api-access-st9ph\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.717898 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.717868    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:13.785745 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:13.785663    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:11:14.094109 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:14.094080    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"]
Jul 08 08:11:14.095186 ip-10-0-140-110 kubenswrapper[2582]: W0708 08:11:14.095123    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2cc767_a049_4234_b165_4a0ce5f7bc37.slice/crio-3ac601dae55c331232d947d9f5360f32d650cb9b70abda4ad8d1e8e49687eec6 WatchSource:0}: Error finding container 3ac601dae55c331232d947d9f5360f32d650cb9b70abda4ad8d1e8e49687eec6: Status 404 returned error can't find the container with id 3ac601dae55c331232d947d9f5360f32d650cb9b70abda4ad8d1e8e49687eec6
Jul 08 08:11:14.162410 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:14.162381    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" event={"ID":"bb2cc767-a049-4234-b165-4a0ce5f7bc37","Type":"ContainerStarted","Data":"dada156ce7d6d6f372ff4a47fc6c2fa8aac35f2a9e25892abb2ca5c213c81ae2"}
Jul 08 08:11:14.162522 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:14.162434    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" event={"ID":"bb2cc767-a049-4234-b165-4a0ce5f7bc37","Type":"ContainerStarted","Data":"3ac601dae55c331232d947d9f5360f32d650cb9b70abda4ad8d1e8e49687eec6"}
Jul 08 08:11:15.022901 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:15.022860    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 08 08:11:18.188937 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:18.188907    2582 generic.go:356] "Generic (PLEG): container finished" podID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerID="dada156ce7d6d6f372ff4a47fc6c2fa8aac35f2a9e25892abb2ca5c213c81ae2" exitCode=0
Jul 08 08:11:18.189364 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:18.188979    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" event={"ID":"bb2cc767-a049-4234-b165-4a0ce5f7bc37","Type":"ContainerDied","Data":"dada156ce7d6d6f372ff4a47fc6c2fa8aac35f2a9e25892abb2ca5c213c81ae2"}
Jul 08 08:11:19.197177 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:19.197101    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" event={"ID":"bb2cc767-a049-4234-b165-4a0ce5f7bc37","Type":"ContainerStarted","Data":"74cb081171f844db3f5c406794ceee8be30885e480f10c61c40c096191630a45"}
Jul 08 08:11:19.224321 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:19.224272    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" podStartSLOduration=2.211645952 podStartE2EDuration="6.224257291s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.012611339s" isStatefulPod=false podCreationTimestamp="2026-07-08 08:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 08:11:19.222576106 +0000 UTC m=+4595.026362545" watchObservedRunningTime="2026-07-08 08:11:19.224257291 +0000 UTC m=+4595.028043690"
Jul 08 08:11:23.718663 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:23.718620    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:23.719096 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:23.718708    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:11:23.720002 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:23.719968    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 08 08:11:25.023134 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:25.023068    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 08 08:11:33.718895 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:33.718840    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 08 08:11:35.022658 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:35.022612    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 08 08:11:43.719193 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:43.719124    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 08 08:11:45.023587 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:45.023486    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 08 08:11:53.718392 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:53.718345    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 08 08:11:55.035319 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:55.035283    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:11:55.044428 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:11:55.044397    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:12:03.718880 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:03.718829    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 08 08:12:12.416121 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:12.416078    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:12:13.719209 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:13.719144    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 08 08:12:21.077181 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:21.072516    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"]
Jul 08 08:12:21.194530 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:21.194495    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:12:21.809504 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:21.809463    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:12:21.914743 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:21.914712    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:12:23.719197 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:23.719125    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 08 08:12:33.718267 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:33.718216    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 08 08:12:36.103963 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:36.103916    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" containerID="cri-o://885073ef47e4a2079bc863e8de147388f84ab417e8c347374a78816325c4055a" gracePeriod=45
Jul 08 08:12:36.427775 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:36.427730    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:12:36.759886 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:36.759787    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:12:37.037495 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.037391    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 08 08:12:37.212046 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.212018    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:12:37.371051 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.370800    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kt5r\" (UniqueName: \"kubernetes.io/projected/72ec3281-e757-40fc-894c-131af6299600-kube-api-access-5kt5r\") pod \"72ec3281-e757-40fc-894c-131af6299600\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") "
Jul 08 08:12:37.371051 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.370876    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-home\") pod \"72ec3281-e757-40fc-894c-131af6299600\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") "
Jul 08 08:12:37.371051 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.370972    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-kserve-provision-location\") pod \"72ec3281-e757-40fc-894c-131af6299600\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") "
Jul 08 08:12:37.371051 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.371043    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-dshm\") pod \"72ec3281-e757-40fc-894c-131af6299600\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") "
Jul 08 08:12:37.371439 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.371094    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ec3281-e757-40fc-894c-131af6299600-tls-certs\") pod \"72ec3281-e757-40fc-894c-131af6299600\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") "
Jul 08 08:12:37.371439 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.371127    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-model-cache\") pod \"72ec3281-e757-40fc-894c-131af6299600\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") "
Jul 08 08:12:37.371439 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.371174    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-tmp-dir\") pod \"72ec3281-e757-40fc-894c-131af6299600\" (UID: \"72ec3281-e757-40fc-894c-131af6299600\") "
Jul 08 08:12:37.371607 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.371546    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-home" (OuterVolumeSpecName: "home") pod "72ec3281-e757-40fc-894c-131af6299600" (UID: "72ec3281-e757-40fc-894c-131af6299600"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 08:12:37.372938 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.371703    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-model-cache" (OuterVolumeSpecName: "model-cache") pod "72ec3281-e757-40fc-894c-131af6299600" (UID: "72ec3281-e757-40fc-894c-131af6299600"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 08:12:37.374391 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.374352    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-dshm" (OuterVolumeSpecName: "dshm") pod "72ec3281-e757-40fc-894c-131af6299600" (UID: "72ec3281-e757-40fc-894c-131af6299600"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 08:12:37.374872 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.374837    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ec3281-e757-40fc-894c-131af6299600-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72ec3281-e757-40fc-894c-131af6299600" (UID: "72ec3281-e757-40fc-894c-131af6299600"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 08:12:37.389718 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.389469    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ec3281-e757-40fc-894c-131af6299600-kube-api-access-5kt5r" (OuterVolumeSpecName: "kube-api-access-5kt5r") pod "72ec3281-e757-40fc-894c-131af6299600" (UID: "72ec3281-e757-40fc-894c-131af6299600"). InnerVolumeSpecName "kube-api-access-5kt5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 08:12:37.416892 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.416839    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72ec3281-e757-40fc-894c-131af6299600" (UID: "72ec3281-e757-40fc-894c-131af6299600"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 08:12:37.422864 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.422824    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "72ec3281-e757-40fc-894c-131af6299600" (UID: "72ec3281-e757-40fc-894c-131af6299600"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 08:12:37.473399 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.473258    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 08:12:37.473399 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.473299    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 08:12:37.473399 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.473316    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ec3281-e757-40fc-894c-131af6299600-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 08:12:37.473399 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.473330    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 08:12:37.473399 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.473343    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 08:12:37.473399 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.473358    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kt5r\" (UniqueName: \"kubernetes.io/projected/72ec3281-e757-40fc-894c-131af6299600-kube-api-access-5kt5r\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 08:12:37.473399 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.473371    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ec3281-e757-40fc-894c-131af6299600-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 08:12:37.650681 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.650596    2582 generic.go:356] "Generic (PLEG): container finished" podID="72ec3281-e757-40fc-894c-131af6299600" containerID="885073ef47e4a2079bc863e8de147388f84ab417e8c347374a78816325c4055a" exitCode=0
Jul 08 08:12:37.650681 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.650662    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"
Jul 08 08:12:37.651070 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.650676    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" event={"ID":"72ec3281-e757-40fc-894c-131af6299600","Type":"ContainerDied","Data":"885073ef47e4a2079bc863e8de147388f84ab417e8c347374a78816325c4055a"}
Jul 08 08:12:37.651070 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.650742    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p" event={"ID":"72ec3281-e757-40fc-894c-131af6299600","Type":"ContainerDied","Data":"a445b096e884290fe8ccd2a9f6cc8eb7fce0829e6c8d87e67e987634a2980cbe"}
Jul 08 08:12:37.651070 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.650775    2582 scope.go:117] "RemoveContainer" containerID="885073ef47e4a2079bc863e8de147388f84ab417e8c347374a78816325c4055a"
Jul 08 08:12:37.669612 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.669588    2582 scope.go:117] "RemoveContainer" containerID="c53365d3781b92cd871a64f5666e0d3221b567905062a9f5096427bf016c6b65"
Jul 08 08:12:37.684445 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.684389    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"]
Jul 08 08:12:37.687893 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.687870    2582 scope.go:117] "RemoveContainer" containerID="885073ef47e4a2079bc863e8de147388f84ab417e8c347374a78816325c4055a"
Jul 08 08:12:37.688860 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.688836    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7854d477c-n9w7p"]
Jul 08 08:12:37.689124 ip-10-0-140-110 kubenswrapper[2582]: E0708 08:12:37.689096    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885073ef47e4a2079bc863e8de147388f84ab417e8c347374a78816325c4055a\": container with ID starting with 885073ef47e4a2079bc863e8de147388f84ab417e8c347374a78816325c4055a not found: ID does not exist" containerID="885073ef47e4a2079bc863e8de147388f84ab417e8c347374a78816325c4055a"
Jul 08 08:12:37.689238 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.689135    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885073ef47e4a2079bc863e8de147388f84ab417e8c347374a78816325c4055a"} err="failed to get container status \"885073ef47e4a2079bc863e8de147388f84ab417e8c347374a78816325c4055a\": rpc error: code = NotFound desc = could not find container \"885073ef47e4a2079bc863e8de147388f84ab417e8c347374a78816325c4055a\": container with ID starting with 885073ef47e4a2079bc863e8de147388f84ab417e8c347374a78816325c4055a not found: ID does not exist"
Jul 08 08:12:37.689238 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.689174    2582 scope.go:117] "RemoveContainer" containerID="c53365d3781b92cd871a64f5666e0d3221b567905062a9f5096427bf016c6b65"
Jul 08 08:12:37.689525 ip-10-0-140-110 kubenswrapper[2582]: E0708 08:12:37.689495    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c53365d3781b92cd871a64f5666e0d3221b567905062a9f5096427bf016c6b65\": container with ID starting with c53365d3781b92cd871a64f5666e0d3221b567905062a9f5096427bf016c6b65 not found: ID does not exist" containerID="c53365d3781b92cd871a64f5666e0d3221b567905062a9f5096427bf016c6b65"
Jul 08 08:12:37.689604 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:37.689531    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53365d3781b92cd871a64f5666e0d3221b567905062a9f5096427bf016c6b65"} err="failed to get container status \"c53365d3781b92cd871a64f5666e0d3221b567905062a9f5096427bf016c6b65\": rpc error: code = NotFound desc = could not find container \"c53365d3781b92cd871a64f5666e0d3221b567905062a9f5096427bf016c6b65\": container with ID starting with c53365d3781b92cd871a64f5666e0d3221b567905062a9f5096427bf016c6b65 not found: ID does not exist"
Jul 08 08:12:38.748738 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:38.748695    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ec3281-e757-40fc-894c-131af6299600" path="/var/lib/kubelet/pods/72ec3281-e757-40fc-894c-131af6299600/volumes"
Jul 08 08:12:43.718650 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:43.718604    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 08 08:12:53.718942 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:12:53.718893    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 08 08:13:03.719021 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:13:03.718977    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 08 08:13:10.648412 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:13:10.648375    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:13:13.718625 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:13:13.718583    2582 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 08 08:13:23.729008 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:13:23.728969    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:13:23.737490 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:13:23.737468    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:14:00.598755 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:00.598724    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:14:20.767859 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:20.767713    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"]
Jul 08 08:14:20.770043 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:20.769472    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:14:20.933878 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:20.933839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:14:34.254496 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:34.254454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:14:35.806911 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:35.806868    2582 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main" containerID="cri-o://74cb081171f844db3f5c406794ceee8be30885e480f10c61c40c096191630a45" gracePeriod=45
Jul 08 08:14:36.617769 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.617744    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:14:36.708990 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.708961    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2cc767-a049-4234-b165-4a0ce5f7bc37-tls-certs\") pod \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") "
Jul 08 08:14:36.709172 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.709051    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-tmp-dir\") pod \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") "
Jul 08 08:14:36.709172 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.709088    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-kserve-provision-location\") pod \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") "
Jul 08 08:14:36.709172 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.709140    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-model-cache\") pod \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") "
Jul 08 08:14:36.709346 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.709201    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-dshm\") pod \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") "
Jul 08 08:14:36.709346 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.709238    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st9ph\" (UniqueName: \"kubernetes.io/projected/bb2cc767-a049-4234-b165-4a0ce5f7bc37-kube-api-access-st9ph\") pod \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") "
Jul 08 08:14:36.709346 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.709266    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-home\") pod \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\" (UID: \"bb2cc767-a049-4234-b165-4a0ce5f7bc37\") "
Jul 08 08:14:36.709512 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.709486    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb2cc767-a049-4234-b165-4a0ce5f7bc37" (UID: "bb2cc767-a049-4234-b165-4a0ce5f7bc37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 08:14:36.709623 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.709607    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-model-cache\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 08:14:36.710233 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.710203    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-home" (OuterVolumeSpecName: "home") pod "bb2cc767-a049-4234-b165-4a0ce5f7bc37" (UID: "bb2cc767-a049-4234-b165-4a0ce5f7bc37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 08:14:36.711653 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.711620    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-dshm" (OuterVolumeSpecName: "dshm") pod "bb2cc767-a049-4234-b165-4a0ce5f7bc37" (UID: "bb2cc767-a049-4234-b165-4a0ce5f7bc37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 08:14:36.711762 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.711675    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2cc767-a049-4234-b165-4a0ce5f7bc37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb2cc767-a049-4234-b165-4a0ce5f7bc37" (UID: "bb2cc767-a049-4234-b165-4a0ce5f7bc37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 08:14:36.711859 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.711836    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2cc767-a049-4234-b165-4a0ce5f7bc37-kube-api-access-st9ph" (OuterVolumeSpecName: "kube-api-access-st9ph") pod "bb2cc767-a049-4234-b165-4a0ce5f7bc37" (UID: "bb2cc767-a049-4234-b165-4a0ce5f7bc37"). InnerVolumeSpecName "kube-api-access-st9ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 08:14:36.728970 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.728939    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb2cc767-a049-4234-b165-4a0ce5f7bc37" (UID: "bb2cc767-a049-4234-b165-4a0ce5f7bc37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 08:14:36.750059 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.750034    2582 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb2cc767-a049-4234-b165-4a0ce5f7bc37" (UID: "bb2cc767-a049-4234-b165-4a0ce5f7bc37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 08:14:36.810701 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.810679    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2cc767-a049-4234-b165-4a0ce5f7bc37-tls-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 08:14:36.810701 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.810701    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-tmp-dir\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 08:14:36.811015 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.810710    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-kserve-provision-location\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 08:14:36.811015 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.810718    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-dshm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 08:14:36.811015 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.810728    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st9ph\" (UniqueName: \"kubernetes.io/projected/bb2cc767-a049-4234-b165-4a0ce5f7bc37-kube-api-access-st9ph\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 08:14:36.811015 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:36.810737    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2cc767-a049-4234-b165-4a0ce5f7bc37-home\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\""
Jul 08 08:14:37.305778 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:37.305748    2582 generic.go:356] "Generic (PLEG): container finished" podID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerID="74cb081171f844db3f5c406794ceee8be30885e480f10c61c40c096191630a45" exitCode=0
Jul 08 08:14:37.305985 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:37.305798    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"
Jul 08 08:14:37.305985 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:37.305836    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" event={"ID":"bb2cc767-a049-4234-b165-4a0ce5f7bc37","Type":"ContainerDied","Data":"74cb081171f844db3f5c406794ceee8be30885e480f10c61c40c096191630a45"}
Jul 08 08:14:37.305985 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:37.305906    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p" event={"ID":"bb2cc767-a049-4234-b165-4a0ce5f7bc37","Type":"ContainerDied","Data":"3ac601dae55c331232d947d9f5360f32d650cb9b70abda4ad8d1e8e49687eec6"}
Jul 08 08:14:37.305985 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:37.305936    2582 scope.go:117] "RemoveContainer" containerID="74cb081171f844db3f5c406794ceee8be30885e480f10c61c40c096191630a45"
Jul 08 08:14:37.316493 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:37.316477    2582 scope.go:117] "RemoveContainer" containerID="dada156ce7d6d6f372ff4a47fc6c2fa8aac35f2a9e25892abb2ca5c213c81ae2"
Jul 08 08:14:37.325619 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:37.325599    2582 scope.go:117] "RemoveContainer" containerID="74cb081171f844db3f5c406794ceee8be30885e480f10c61c40c096191630a45"
Jul 08 08:14:37.325917 ip-10-0-140-110 kubenswrapper[2582]: E0708 08:14:37.325893    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74cb081171f844db3f5c406794ceee8be30885e480f10c61c40c096191630a45\": container with ID starting with 74cb081171f844db3f5c406794ceee8be30885e480f10c61c40c096191630a45 not found: ID does not exist" containerID="74cb081171f844db3f5c406794ceee8be30885e480f10c61c40c096191630a45"
Jul 08 08:14:37.325996 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:37.325922    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74cb081171f844db3f5c406794ceee8be30885e480f10c61c40c096191630a45"} err="failed to get container status \"74cb081171f844db3f5c406794ceee8be30885e480f10c61c40c096191630a45\": rpc error: code = NotFound desc = could not find container \"74cb081171f844db3f5c406794ceee8be30885e480f10c61c40c096191630a45\": container with ID starting with 74cb081171f844db3f5c406794ceee8be30885e480f10c61c40c096191630a45 not found: ID does not exist"
Jul 08 08:14:37.325996 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:37.325940    2582 scope.go:117] "RemoveContainer" containerID="dada156ce7d6d6f372ff4a47fc6c2fa8aac35f2a9e25892abb2ca5c213c81ae2"
Jul 08 08:14:37.326214 ip-10-0-140-110 kubenswrapper[2582]: E0708 08:14:37.326198    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dada156ce7d6d6f372ff4a47fc6c2fa8aac35f2a9e25892abb2ca5c213c81ae2\": container with ID starting with dada156ce7d6d6f372ff4a47fc6c2fa8aac35f2a9e25892abb2ca5c213c81ae2 not found: ID does not exist" containerID="dada156ce7d6d6f372ff4a47fc6c2fa8aac35f2a9e25892abb2ca5c213c81ae2"
Jul 08 08:14:37.326268 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:37.326218    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dada156ce7d6d6f372ff4a47fc6c2fa8aac35f2a9e25892abb2ca5c213c81ae2"} err="failed to get container status \"dada156ce7d6d6f372ff4a47fc6c2fa8aac35f2a9e25892abb2ca5c213c81ae2\": rpc error: code = NotFound desc = could not find container \"dada156ce7d6d6f372ff4a47fc6c2fa8aac35f2a9e25892abb2ca5c213c81ae2\": container with ID starting with dada156ce7d6d6f372ff4a47fc6c2fa8aac35f2a9e25892abb2ca5c213c81ae2 not found: ID does not exist"
Jul 08 08:14:37.336290 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:37.336263    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"]
Jul 08 08:14:37.341922 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:37.341899    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79h9b8p"]
Jul 08 08:14:38.748440 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:38.748400    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" path="/var/lib/kubelet/pods/bb2cc767-a049-4234-b165-4a0ce5f7bc37/volumes"
Jul 08 08:14:45.087821 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:45.087742    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:14:45.102543 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:45.102514    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:14:46.697279 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:46.697246    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:14:57.526128 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:57.526097    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:14:57.805781 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:14:57.805668    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:15:23.799061 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:15:23.799030    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:15:27.597464 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:15:27.597416    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:15:27.697488 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:15:27.697438    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:15:48.612728 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:15:48.612649    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:15:48.713550 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:15:48.713496    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:16:03.337401 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:16:03.337348    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:17:26.081433 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:17:26.081350    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:18:07.206786 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:18:07.206745    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:18:07.300098 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:18:07.300062    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:18:26.998066 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:18:26.998024    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:18:28.095549 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:18:28.095512    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:18:28.179642 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:18:28.179589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:18:31.628984 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:18:31.628949    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:19:02.599542 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:19:02.599502    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:19:03.365510 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:19:03.365463    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:19:19.269619 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:19:19.269586    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:19:25.336051 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:19:25.336012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:19:45.119213 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:19:45.119113    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:19:45.134326 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:19:45.134299    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:20:49.289034 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:20:49.288993    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:21:28.374923 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:21:28.374890    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:21:39.605542 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:21:39.605436    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:22:39.642496 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:22:39.642457    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:23:19.841568 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:23:19.841480    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:23:20.150542 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:23:20.150462    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:23:43.336631 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:23:43.336589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:24:04.015559 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:24:04.015526    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:24:45.148024 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:24:45.147997    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:24:45.163635 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:24:45.163614    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:25:35.730262 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:25:35.730220    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:25:35.793817 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:25:35.793783    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:26:07.416322 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:26:07.416216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:26:08.671408 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:26:08.671372    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:26:31.411141 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:26:31.411094    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:28:40.553140 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:28:40.553039    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:28:40.623516 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:28:40.623477    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:29:33.308586 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:29:33.308456    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:29:45.176951 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:29:45.176926    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:29:45.193272 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:29:45.193250    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:29:51.943477 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:29:51.943441    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:29:52.257043 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:29:52.256934    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:30:18.131602 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:30:18.131549    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:30:21.668178 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:30:21.668128    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:30:43.611662 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:30:43.611625    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:30:43.710888 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:30:43.710821    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:31:32.535917 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:31:32.535870    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:33:37.589786 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:37.589703    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:33:37.679519 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:37.679484    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:33:55.308277 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:55.308242    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:33:55.318057 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:55.318032    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:33:55.335890 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:55.335868    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:33:55.349926 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:55.349903    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:33:55.416460 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:55.416433    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:33:55.423898 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:55.423876    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:33:55.575273 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:55.575190    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:33:55.584255 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:55.584231    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:33:55.684026 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:55.683994    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:33:55.694678 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:55.694650    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:33:55.815613 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:55.815585    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:33:55.824006 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:55.823984    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:33:56.490002 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:56.489972    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:33:56.502445 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:56.502423    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:33:56.767086 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:56.767006    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:33:56.774124 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:56.774097    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:33:58.575664 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:58.575625    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:33:58.596604 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:58.596586    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:33:58.634231 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:58.634201    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:33:58.663104 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:58.663081    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:33:58.744145 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:58.744114    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:33:58.757651 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:58.757630    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:33:58.916956 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:58.916930    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:33:58.927292 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:58.927269    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:33:59.028772 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:59.028736    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:33:59.037079 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:59.037052    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:33:59.144641 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:59.144610    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:33:59.152302 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:59.152278    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:33:59.851800 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:59.851765    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:33:59.871679 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:33:59.871646    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:00.119355 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:00.119278    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:00.127849 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:00.127815    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:01.949110 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:01.949074    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:01.957947 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:01.957916    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:01.977736 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:01.977712    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:01.991560 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:01.991542    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:02.064034 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:02.064007    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:02.072233 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:02.072211    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:02.208616 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:02.208530    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:02.216031 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:02.216005    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:02.317794 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:02.317761    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:02.326307 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:02.326282    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:02.447947 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:02.447919    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:02.455999 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:02.455977    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:03.160222 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:03.160187    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:03.176124 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:03.176099    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:03.414911 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:03.414888    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:03.423037 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:03.423011    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:05.233625 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:05.233591    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:05.243646 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:05.243623    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:05.263528 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:05.263507    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:05.276676 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:05.276658    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:05.347134 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:05.347112    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:05.362910 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:05.362891    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:05.494805 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:05.494725    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:05.529130 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:05.529108    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:05.625630 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:05.625602    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:05.640937 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:05.640910    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:05.758501 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:05.758421    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:05.766711 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:05.766691    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:06.465072 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:06.465034    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:06.544112 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:06.544086    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:06.796688 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:06.796597    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:06.810235 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:06.810209    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:08.609294 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:08.609263    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:08.617109 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:08.617088    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:08.637146 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:08.637122    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:08.649040 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:08.649014    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:08.726340 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:08.726308    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:08.734541 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:08.734510    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:08.876220 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:08.876106    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:08.884572 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:08.884543    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:08.996476 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:08.996445    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:09.004491 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:09.004461    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:09.121507 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:09.121480    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:09.151426 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:09.151353    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:09.784463 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:09.784426    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:09.801276 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:09.801249    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:10.054068 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:10.053996    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:10.061115 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:10.061076    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:11.878654 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:11.878613    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:11.887717 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:11.887689    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:11.906715 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:11.906687    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:11.930169 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:11.930129    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:12.002825 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:12.002793    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:12.010660 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:12.010638    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:12.159893 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:12.159747    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:12.171196 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:12.171172    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:12.264946 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:12.264911    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:12.274050 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:12.274022    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:12.408372 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:12.408335    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:12.418604 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:12.418579    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:13.128547 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:13.128508    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:13.147963 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:13.147923    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:13.397953 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:13.397878    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:13.430939 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:13.430914    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:15.234608 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:15.234572    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:15.243317 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:15.243287    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:15.267812 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:15.267789    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:15.281649 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:15.281626    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:15.350544 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:15.350514    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:15.358315 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:15.358295    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:15.494637 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:15.494557    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:15.504801 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:15.504772    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:15.594625 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:15.594600    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:15.604877 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:15.604846    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:15.732190 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:15.732147    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:15.740263 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:15.740239    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:16.501851 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:16.501817    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:16.520265 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:16.520235    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:16.773855 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:16.773769    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:16.781992 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:16.781962    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:18.592320 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:18.592280    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:18.599708 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:18.599683    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:18.617013 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:18.616986    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:18.630071 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:18.630047    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:18.703242 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:18.703200    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:18.710768 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:18.710740    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:18.841665 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:18.841633    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:18.853354 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:18.853279    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:18.944710 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:18.944673    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:18.952190 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:18.952147    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:19.079440 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:19.079402    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:19.087046 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:19.087008    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:19.787265 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:19.787226    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:19.805536 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:19.805507    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:20.043037 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:20.042946    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:20.054680 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:20.054649    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:21.889640 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:21.889605    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:21.899682 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:21.899642    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:21.919735 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:21.919707    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:21.932473 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:21.932442    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:22.004655 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:22.004609    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:22.013414 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:22.013387    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:22.174980 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:22.174940    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:22.184352 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:22.184324    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:22.282955 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:22.282917    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:22.290129 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:22.290108    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:22.417789 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:22.417755    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:22.426008 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:22.425924    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:23.121508 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:23.121469    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:23.147123 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:23.147091    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:23.410194 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:23.410087    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:23.418939 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:23.418913    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:25.144022 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:25.143975    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:25.152415 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:25.152389    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:25.172021 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:25.171986    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:25.184029 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:25.184002    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:25.250453 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:25.250412    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:25.258391 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:25.258342    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:25.401976 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:25.401877    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:25.410073 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:25.410050    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:25.512085 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:25.511938    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:25.519593 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:25.519554    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:25.636564 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:25.636519    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:25.646644 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:25.646619    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:26.314779 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:26.314746    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:26.326766 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:26.326718    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:26.562806 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:26.562769    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:26.569755 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:26.569690    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:28.316592 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:28.316563    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:28.324404 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:28.324377    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:28.344793 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:28.344770    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:28.358114 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:28.358094    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:28.432396 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:28.432365    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:28.440079 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:28.440051    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:28.576910 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:28.576806    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:28.583684 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:28.583660    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:28.695604 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:28.695569    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:28.703593 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:28.703565    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:28.822871 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:28.822840    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:28.830625 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:28.830521    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:29.468254 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:29.468214    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:29.484288 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:29.484259    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:29.720276 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:29.720187    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:29.727820 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:29.727797    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:31.441756 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:31.441722    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:31.449730 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:31.449701    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:31.472329 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:31.472297    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:31.486689 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:31.486663    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:31.563033 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:31.563008    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:31.573023 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:31.572995    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:31.724455 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:31.724368    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:31.731889 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:31.731863    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:31.839880 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:31.839850    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:31.846884 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:31.846862    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:31.952683 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:31.952657    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:31.960473 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:31.960454    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:32.585414 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:32.585378    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:32.605112 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:32.605090    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:32.853922 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:32.853827    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:32.861354 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:32.861329    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:34.664939 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:34.664881    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:34.673419 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:34.673368    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:34.691622 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:34.691593    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:34.704175 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:34.704126    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:34.789087 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:34.789063    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:34.796319 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:34.796296    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:34.926097 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:34.926015    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:34.934992 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:34.934966    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:35.034142 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:35.034113    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:35.041559 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:35.041534    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:35.146739 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:35.146717    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:35.158441 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:35.158419    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:35.816880 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:35.816847    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:35.840859 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:35.840833    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:36.106976 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:36.106858    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:36.116355 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:36.116331    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:37.887103 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:37.887075    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:37.899908 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:37.899881    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:37.927013 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:37.926979    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:37.943879 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:37.943835    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:38.019161 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:38.019122    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:38.029498 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:38.029478    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:38.171088 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:38.171064    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:38.190128 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:38.190107    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:38.286686 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:38.286655    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:38.296721 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:38.296698    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:38.418920 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:38.418893    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:38.479917 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:38.479834    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:39.197507 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:39.197472    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:39.219060 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:39.219034    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:39.457668 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:39.457585    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:39.465066 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:39.465040    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:41.249050 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:41.249025    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:41.257906 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:41.257884    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:41.278603 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:41.278574    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:41.310145 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:41.310115    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:41.388798 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:41.388769    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:41.396315 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:41.396294    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:41.546462 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:41.546363    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:41.555726 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:41.555703    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:41.648428 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:41.648401    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:41.686248 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:41.686222    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:41.836575 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:41.836487    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:41.850889 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:41.850863    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:42.586670 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:42.586635    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:42.607081 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:42.607054    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:42.871197 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:42.871094    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:42.881342 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:42.881318    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:44.623070 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:44.623034    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:44.630443 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:44.630401    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:44.647074 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:44.647054    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:44.660800 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:44.660780    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:44.735650 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:44.735627    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:44.749625 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:44.749595    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:44.881296 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:44.881213    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:44.889579 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:44.889554    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:44.989062 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:44.989026    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:44.997834 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:44.997811    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:45.115979 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:45.115943    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:45.123787 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:45.123759    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:45.217083 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:45.216929    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:34:45.231133 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:45.231107    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:34:45.808451 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:45.808406    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:45.826253 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:45.826192    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:46.059002 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:46.058929    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:46.067147 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:46.067125    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:48.372903 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:48.372852    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-g22cx_c1e1a16a-a06e-4b84-842c-6b4415e0f636/istio-proxy/11.log"
Jul 08 08:34:48.397251 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:48.397213    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-g22cx_c1e1a16a-a06e-4b84-842c-6b4415e0f636/istio-proxy/12.log"
Jul 08 08:34:49.451258 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:49.451231    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:49.462188 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:49.462145    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:49.477264 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:49.477241    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:49.489720 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:49.489701    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:49.560837 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:49.560811    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:49.567838 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:49.567810    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:49.715694 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:49.715589    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:49.723260 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:49.723237    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:49.828909 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:49.828499    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:49.835495 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:49.835471    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:49.945562 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:49.945532    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:49.953799 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:49.953770    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:50.596578 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:50.596544    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:50.615804 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:50.615766    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:50.873687 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:50.873606    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:50.880760 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:50.880735    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:52.688577 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:52.688542    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:52.696645 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:52.696612    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:52.716496 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:52.716476    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:52.729028 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:52.729002    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:52.800178 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:52.800136    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:52.808867 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:52.808848    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:52.961887 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:52.961788    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:52.980750 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:52.980724    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:53.094983 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:53.094946    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:53.111874 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:53.111851    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:53.267897 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:53.267805    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:53.277730 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:53.277704    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:53.920939 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:53.920911    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:53.942054 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:53.942030    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:54.224586 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:54.224512    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:54.234086 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:54.234059    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:56.057879 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:56.056824    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:56.065249 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:56.065218    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:56.083125 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:56.083099    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:56.097061 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:56.097041    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:56.179549 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:56.179514    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:56.187061 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:56.187037    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:56.319249 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:56.319148    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:56.326808 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:56.326784    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:56.418803 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:56.418777    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:56.428014 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:56.427990    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:56.531205 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:56.531182    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:56.539108 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:56.539088    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:34:57.180395 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:57.180359    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:34:57.192787 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:57.192762    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:34:57.424571 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:57.424539    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:34:57.433797 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:57.433713    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:34:59.225861 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:59.225828    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:34:59.233567 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:59.233545    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:34:59.250666 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:59.250645    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:34:59.262307 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:59.262286    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:34:59.333408 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:59.333328    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:34:59.341068 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:59.341043    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:34:59.465580 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:59.465548    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:34:59.473418 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:59.473395    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:34:59.566853 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:59.566777    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:34:59.573835 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:59.573806    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:34:59.671108 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:59.671084    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:34:59.695124 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:34:59.695101    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:35:00.402328 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:00.402288    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:35:00.421604 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:00.421573    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:35:00.718111 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:00.718067    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:35:00.726663 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:00.726629    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:35:02.593044 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:02.593005    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:35:02.611582 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:02.611557    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:35:02.642347 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:02.642323    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:35:02.664593 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:02.664570    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:35:02.745754 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:02.745726    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:35:02.757548 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:02.757527    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:35:02.914881 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:02.914843    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:35:02.924064 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:02.924043    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:35:03.015690 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:03.015670    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:35:03.029915 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:03.029891    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:35:03.156311 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:03.156279    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:35:03.174218 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:03.174100    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:35:03.923603 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:03.923569    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:35:03.943755 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:03.943724    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:35:04.192377 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:04.192266    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:35:04.202934 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:04.202908    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:35:05.988073 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:05.988041    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/main/0.log"
Jul 08 08:35:05.995558 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:05.995534    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-68f5d8959c-xh45g_275c2425-5390-4987-804a-f70b52e9dd60/storage-initializer/0.log"
Jul 08 08:35:06.013080 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:06.013059    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/main/0.log"
Jul 08 08:35:06.024945 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:06.024918    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-disabled-test-kserve-router-scheduler-5898f785fb-xr25g_b0f58936-1d64-48ee-90e4-40b8b1145eeb/storage-initializer/0.log"
Jul 08 08:35:06.098921 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:06.098897    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/main/0.log"
Jul 08 08:35:06.106560 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:06.106541    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-b6f9dd5f7-llmnq_552f4e64-1b67-44ee-adf4-56fdad788beb/storage-initializer/0.log"
Jul 08 08:35:06.242388 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:06.242301    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/main/0.log"
Jul 08 08:35:06.259951 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:06.259926    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c57b59c67dgldz_eda082cc-9a14-4867-8ac5-19d540f8ec41/storage-initializer/0.log"
Jul 08 08:35:06.353133 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:06.353100    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/main/0.log"
Jul 08 08:35:06.361613 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:06.361587    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6694784b54-grkws_ec1d0b1a-7cd1-49d9-b280-796bd8f53eca/storage-initializer/0.log"
Jul 08 08:35:06.475509 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:06.475483    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/main/0.log"
Jul 08 08:35:06.483219 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:06.483186    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-55585bf4d9-8bdd6_57b1e783-1750-4f22-8ebd-bc305536de13/storage-initializer/0.log"
Jul 08 08:35:07.219579 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:07.219478    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0_9d37b6b1-ed42-4a7e-aa20-371c401b8c99/main/0.log"
Jul 08 08:35:07.239084 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:07.239054    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1_48d4f055-cfec-400b-ac7c-666b216ccaca/main/0.log"
Jul 08 08:35:07.473473 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:07.473398    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/main/0.log"
Jul 08 08:35:07.481135 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:07.481105    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-794b6f7745-fpslh_df3d2e0b-ce6b-4275-b4f5-dae9bf0960e9/storage-initializer/0.log"
Jul 08 08:35:09.792805 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:09.792779    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-g22cx_c1e1a16a-a06e-4b84-842c-6b4415e0f636/istio-proxy/11.log"
Jul 08 08:35:09.811094 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:09.811072    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-g22cx_c1e1a16a-a06e-4b84-842c-6b4415e0f636/istio-proxy/12.log"
Jul 08 08:35:11.413469 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:11.413410    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-g22cx_c1e1a16a-a06e-4b84-842c-6b4415e0f636/istio-proxy/11.log"
Jul 08 08:35:11.434696 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:11.434660    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-g22cx_c1e1a16a-a06e-4b84-842c-6b4415e0f636/istio-proxy/12.log"
Jul 08 08:35:12.986946 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:12.986911    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-g22cx_c1e1a16a-a06e-4b84-842c-6b4415e0f636/istio-proxy/11.log"
Jul 08 08:35:13.008499 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:13.008453    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-g22cx_c1e1a16a-a06e-4b84-842c-6b4415e0f636/istio-proxy/12.log"
Jul 08 08:35:14.029008 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:14.028981    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:14.078062 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:14.078026    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:15.457906 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:15.457875    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:15.513306 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:15.513280    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:16.914959 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:16.914930    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:16.956773 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:16.956751    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:18.457406 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:18.457374    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:18.505568 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:18.505537    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:19.876433 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:19.876393    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:19.923916 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:19.923894    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:21.363207 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:21.363147    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:21.417567 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:21.417544    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:22.809898 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:22.809834    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:22.873057 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:22.873024    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:24.326538 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:24.326508    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:24.374924 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:24.374896    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:25.745086 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:25.745055    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:25.788487 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:25.788461    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:27.139381 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:27.139351    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:27.181998 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:27.181972    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:28.542203 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:28.542171    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:28.584739 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:28.584711    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:30.042617 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:30.042585    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:30.090724 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:30.090696    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:31.450072 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:31.450039    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:31.503434 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:31.503408    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:32.885795 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:32.885765    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:32.928693 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:32.928664    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:34.340600 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:34.340557    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:34.386789 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:34.386763    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:35.779636 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:35.779592    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:35.823473 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:35.823446    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:37.205703 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:37.205673    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:37.247917 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:37.247893    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:38.638595 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:38.638560    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:38.682983 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:38.682957    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:40.035856 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:40.035816    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:40.086467 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:40.086441    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:41.540672 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:41.540641    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:41.581016 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:41.580990    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:42.991464 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:42.991425    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:43.034624 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:43.034600    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:44.404688 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:44.404648    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:44.453952 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:44.453926    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:45.846409 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:45.846379    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:45.889638 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:45.889608    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:47.268308 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:47.268273    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:47.316528 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:47.316507    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:48.730975 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:48.730938    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:48.777209 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:48.777180    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:50.148318 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:50.148287    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:50.191746 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:50.191715    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:51.551515 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:51.551479    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:51.600411 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:51.600384    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:52.958161 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:52.958119    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:53.002402 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:53.002377    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:54.392116 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:54.392066    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:54.439245 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:54.439223    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:55.857400 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:55.857362    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:55.903697 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:55.903670    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:57.291664 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:57.291637    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:57.334224 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:57.334200    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:35:58.761086 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:58.761055    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:35:58.803081 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:35:58.803053    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:00.206619 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:00.206581    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:00.251553 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:00.251529    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:01.662468 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:01.662434    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:01.727082 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:01.727056    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:03.231209 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:03.231180    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:03.317684 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:03.317661    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:04.703463 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:04.703431    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:04.744164 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:04.744122    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:06.130708 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:06.130678    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:06.175511 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:06.175483    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:07.544458 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:07.544431    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:07.589781 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:07.589759    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:09.018385 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:09.018350    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:09.062165 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:09.062136    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:10.445263 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:10.445225    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:10.490943 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:10.490911    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:11.873687 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:11.873659    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:11.920214 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:11.920186    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:13.322123 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:13.322093    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:13.386523 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:13.386492    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:14.802895 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:14.802860    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:14.847113 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:14.847083    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:16.246854 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:16.246818    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:16.291338 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:16.291309    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:17.770454 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:17.770429    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:17.819591 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:17.819568    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:19.172748 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:19.172717    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:19.224611 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:19.224587    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:20.577020 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:20.576991    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:20.623191 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:20.623147    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:21.982906 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:21.982880    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:22.029167 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:22.029134    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:23.404700 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:23.404663    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:23.453957 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:23.453933    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:24.870937 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:24.870911    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:24.923880 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:24.923851    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:26.322842 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:26.322810    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:26.369858 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:26.369833    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:27.763413 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:27.763384    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:27.805553 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:27.805514    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:29.198575 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:29.198532    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:29.243768 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:29.243741    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:30.601344 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:30.601309    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:30.652424 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:30.652394    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:32.058565 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:32.058521    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:32.104736 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:32.104713    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:33.500945 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:33.500911    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:33.545371 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:33.545343    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:34.937934 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:34.937891    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:34.981219 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:34.981196    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:36.388091 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:36.388060    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:36.433353 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:36.433330    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:37.853089 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:37.852871    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:37.893960 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:37.893932    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:39.259989 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:39.259955    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:39.304085 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:39.304058    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:40.700335 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:40.700307    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:40.750294 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:40.750270    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:42.178700 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:42.178655    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:42.219965 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:42.219941    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:45.181329 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:45.181293    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:36:46.744613 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:46.744581    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:36:48.553012 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:48.552978    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9snpb_dbf315f9-c635-4d76-81fb-39beef3d0746/global-pull-secret-syncer/0.log"
Jul 08 08:36:48.649265 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:48.649233    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log"
Jul 08 08:36:48.858118 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:48.858025    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-110.ec2.internal_813e38510b3ec379e921946926d00ffe/haproxy/0.log"
Jul 08 08:36:51.175379 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:51.175344    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:36:51.242840 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:51.242806    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gswxv_6a392272-c07e-41e4-997b-af669a4bd16e/konnectivity-agent/0.log" limit=1024
Jul 08 08:36:54.790877 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:54.790839    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-5bxlp_b975a5c2-cb2f-4047-bd0b-a50ef9456b10/authorino/0.log"
Jul 08 08:36:54.932690 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:54.932663    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-8r75k_b5c4102a-0424-46a9-822c-2057ea98f8c9/kuadrant-console-plugin/0.log"
Jul 08 08:36:57.241090 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:57.241051    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4wx47_8e344bb8-5819-4b55-a499-dea024444e95/node-exporter/0.log"
Jul 08 08:36:57.281307 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:57.281267    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4wx47_8e344bb8-5819-4b55-a499-dea024444e95/kube-rbac-proxy/0.log"
Jul 08 08:36:57.312529 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:36:57.312497    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4wx47_8e344bb8-5819-4b55-a499-dea024444e95/init-textfile/0.log"
Jul 08 08:37:00.010816 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.010783    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/1.log"
Jul 08 08:37:00.016683 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.016655    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-nwhbt_c0d34567-03c2-42ac-a6a0-61c6531c483e/console-operator/2.log"
Jul 08 08:37:00.152475 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.152440    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"]
Jul 08 08:37:00.152826 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.152808    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="storage-initializer"
Jul 08 08:37:00.152826 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.152826    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="storage-initializer"
Jul 08 08:37:00.152973 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.152841    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="storage-initializer"
Jul 08 08:37:00.152973 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.152847    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="storage-initializer"
Jul 08 08:37:00.152973 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.152856    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main"
Jul 08 08:37:00.152973 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.152862    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main"
Jul 08 08:37:00.152973 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.152878    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main"
Jul 08 08:37:00.152973 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.152883    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main"
Jul 08 08:37:00.152973 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.152950    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ec3281-e757-40fc-894c-131af6299600" containerName="main"
Jul 08 08:37:00.152973 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.152958    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2cc767-a049-4234-b165-4a0ce5f7bc37" containerName="main"
Jul 08 08:37:00.156109 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.156088    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.158523 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.158499    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cc4px\"/\"default-dockercfg-zzsrs\""
Jul 08 08:37:00.159671 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.159652    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cc4px\"/\"kube-root-ca.crt\""
Jul 08 08:37:00.159731 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.159699    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cc4px\"/\"openshift-service-ca.crt\""
Jul 08 08:37:00.167547 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.167520    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"]
Jul 08 08:37:00.253266 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.253226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/19526441-c853-4ff4-b45a-3d4a10514027-podres\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.253448 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.253324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19526441-c853-4ff4-b45a-3d4a10514027-lib-modules\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.253448 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.253357    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19526441-c853-4ff4-b45a-3d4a10514027-sys\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.253555 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.253471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lggm8\" (UniqueName: \"kubernetes.io/projected/19526441-c853-4ff4-b45a-3d4a10514027-kube-api-access-lggm8\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.253555 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.253524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/19526441-c853-4ff4-b45a-3d4a10514027-proc\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.354807 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.354717    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/19526441-c853-4ff4-b45a-3d4a10514027-podres\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.354807 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.354786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19526441-c853-4ff4-b45a-3d4a10514027-lib-modules\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.355045 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.354822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19526441-c853-4ff4-b45a-3d4a10514027-sys\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.355045 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.354860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lggm8\" (UniqueName: \"kubernetes.io/projected/19526441-c853-4ff4-b45a-3d4a10514027-kube-api-access-lggm8\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.355045 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.354890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/19526441-c853-4ff4-b45a-3d4a10514027-proc\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.355045 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.354912    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/19526441-c853-4ff4-b45a-3d4a10514027-podres\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.355045 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.354922    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19526441-c853-4ff4-b45a-3d4a10514027-lib-modules\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.355045 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.354954    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19526441-c853-4ff4-b45a-3d4a10514027-sys\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.355045 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.355033    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/19526441-c853-4ff4-b45a-3d4a10514027-proc\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.365556 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.365522    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lggm8\" (UniqueName: \"kubernetes.io/projected/19526441-c853-4ff4-b45a-3d4a10514027-kube-api-access-lggm8\") pod \"perf-node-gather-daemonset-nhkx6\" (UID: \"19526441-c853-4ff4-b45a-3d4a10514027\") " pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.467409 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.467377    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:00.622521 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:00.622444    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"]
Jul 08 08:37:00.625634 ip-10-0-140-110 kubenswrapper[2582]: W0708 08:37:00.625608    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod19526441_c853_4ff4_b45a_3d4a10514027.slice/crio-76356dd0c03220a7bdaa3bb1d0f844c99e34c2c4ee27f094a3bbb6444f9965ae WatchSource:0}: Error finding container 76356dd0c03220a7bdaa3bb1d0f844c99e34c2c4ee27f094a3bbb6444f9965ae: Status 404 returned error can't find the container with id 76356dd0c03220a7bdaa3bb1d0f844c99e34c2c4ee27f094a3bbb6444f9965ae
Jul 08 08:37:01.171647 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:01.171609    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6" event={"ID":"19526441-c853-4ff4-b45a-3d4a10514027","Type":"ContainerStarted","Data":"e8ae0466b4ca15a1908f6a3042bfc67c6b31c26f144a24300ad76aa6bf2c7209"}
Jul 08 08:37:01.172206 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:01.171673    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6" event={"ID":"19526441-c853-4ff4-b45a-3d4a10514027","Type":"ContainerStarted","Data":"76356dd0c03220a7bdaa3bb1d0f844c99e34c2c4ee27f094a3bbb6444f9965ae"}
Jul 08 08:37:01.172206 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:01.171806    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:01.197853 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:01.197788    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6" podStartSLOduration=1.197773636 podStartE2EDuration="1.197773636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 08:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 08:37:01.19676179 +0000 UTC m=+6137.000548200" watchObservedRunningTime="2026-07-08 08:37:01.197773636 +0000 UTC m=+6137.001560035"
Jul 08 08:37:01.345305 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:01.345274    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-qb2c9_32c0393b-145c-472b-95cd-267c8bc7457b/volume-data-source-validator/0.log"
Jul 08 08:37:02.651973 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:02.651942    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gjw5t_e142a9e3-7af5-4e3a-9b0e-070f68044486/dns/0.log"
Jul 08 08:37:02.685315 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:02.685294    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gjw5t_e142a9e3-7af5-4e3a-9b0e-070f68044486/kube-rbac-proxy/0.log"
Jul 08 08:37:02.769964 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:02.769938    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6v9zs_8d6fe090-7fa6-4c4c-a339-4f0059c5e6e9/dns-node-resolver/0.log"
Jul 08 08:37:03.491534 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:03.491502    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2tswp_5ae16139-e371-414e-982c-221c0223968a/node-ca/0.log"
Jul 08 08:37:05.375506 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:05.375466    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-g22cx_c1e1a16a-a06e-4b84-842c-6b4415e0f636/istio-proxy/11.log"
Jul 08 08:37:05.400220 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:05.400188    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-g22cx_c1e1a16a-a06e-4b84-842c-6b4415e0f636/istio-proxy/12.log"
Jul 08 08:37:06.101651 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:06.101618    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wxcjq_bc6bc5da-ac3a-43be-a3b9-7468831da20e/serve-healthcheck-canary/0.log"
Jul 08 08:37:06.817908 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:06.817879    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvvnp_3ee5c8ec-83a9-4b40-b2db-25c67997ca4b/kube-rbac-proxy/0.log"
Jul 08 08:37:06.843790 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:06.843763    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvvnp_3ee5c8ec-83a9-4b40-b2db-25c67997ca4b/exporter/0.log"
Jul 08 08:37:06.879657 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:06.879624    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jvvnp_3ee5c8ec-83a9-4b40-b2db-25c67997ca4b/extractor/0.log"
Jul 08 08:37:07.187195 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:07.187142    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cc4px/perf-node-gather-daemonset-nhkx6"
Jul 08 08:37:10.319340 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:10.319304    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58c59b5fc8-lbghs_5a075109-5098-4ab8-8b73-2e85e402401a/manager/0.log"
Jul 08 08:37:11.191655 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:11.191626    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-9487895bf-gwpxw_1bbd95a9-924b-43e2-9803-f0bc0d3d899c/manager/0.log"
Jul 08 08:37:12.223175 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:12.223134    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-szb56_3cb033b1-dc38-4cca-b0a4-95c836e7be0c/seaweedfs/0.log"
Jul 08 08:37:13.122564 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:13.122526    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-g22cx_c1e1a16a-a06e-4b84-842c-6b4415e0f636/istio-proxy/11.log"
Jul 08 08:37:13.141730 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:13.141640    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-g22cx_c1e1a16a-a06e-4b84-842c-6b4415e0f636/istio-proxy/12.log"
Jul 08 08:37:14.891108 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:14.891065    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-g22cx_c1e1a16a-a06e-4b84-842c-6b4415e0f636/istio-proxy/11.log"
Jul 08 08:37:14.912732 ip-10-0-140-110 kubenswrapper[2582]: I0708 08:37:14.912706    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-9455cf8d5-g22cx_c1e1a16a-a06e-4b84-842c-6b4415e0f636/istio-proxy/12.log"