Jul 10 18:38:11.443393 ip-10-0-132-91 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 10 18:38:11.443410 ip-10-0-132-91 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 10 18:38:11.443421 ip-10-0-132-91 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 10 18:38:11.443751 ip-10-0-132-91 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 10 18:38:21.535969 ip-10-0-132-91 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 10 18:38:21.535985 ip-10-0-132-91 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5bfeb62c6fd641558ce244cd2424212a --
Jul 10 18:40:22.412283 ip-10-0-132-91 systemd[1]: Starting Kubernetes Kubelet...
Jul 10 18:40:22.884706 ip-10-0-132-91 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:40:22.884706 ip-10-0-132-91 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 10 18:40:22.884706 ip-10-0-132-91 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:40:22.884706 ip-10-0-132-91 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 10 18:40:22.884706 ip-10-0-132-91 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:40:22.884706 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.810722    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 10 18:40:22.884706 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814784    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:40:22.884706 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814795    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:40:22.884706 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814798    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:40:22.884706 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814801    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:40:22.884706 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814804    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:40:22.884706 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814807    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:40:22.884706 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814810    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:40:22.884706 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814813    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814815    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814818    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814821    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814823    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814825    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814828    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814830    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814832    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814839    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814842    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814844    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814846    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814849    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814851    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814854    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814856    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814858    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814861    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814865    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:40:22.885501 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814869    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814872    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814875    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814877    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814880    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814883    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814885    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814887    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814890    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814892    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814894    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814897    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814900    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814903    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814905    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814908    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814910    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814912    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814916    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:40:22.886075 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814920    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814923    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814926    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814928    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814931    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814934    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814936    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814938    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814941    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814943    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814946    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814949    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814951    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814953    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814955    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814958    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814960    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814972    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814977    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814980    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:40:22.886662 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814983    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814985    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814988    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814990    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.814993    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815001    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815003    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815006    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815008    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815011    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815014    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815016    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815019    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815022    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815024    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815027    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815029    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815031    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815034    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:40:22.887231 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815036    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815039    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815041    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815044    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815046    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815048    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815051    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815053    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815056    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815058    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815067    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815069    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815072    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815074    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815807    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815813    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815816    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815819    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815822    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815831    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:40:22.887808 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815834    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815837    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815839    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815842    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815844    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815848    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815851    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815853    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815856    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815858    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815860    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815863    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815865    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815867    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815870    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815872    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815874    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815877    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815879    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:40:22.888336 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815882    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815884    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815887    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815889    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815892    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815894    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815897    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815899    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815901    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815903    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815906    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815908    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815911    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815914    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815916    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815918    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815921    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815923    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815925    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815928    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:40:22.888918 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815930    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815932    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815934    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815937    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815939    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815942    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815945    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815947    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815949    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815952    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815955    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815957    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815960    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815962    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815964    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815966    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815969    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815972    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815975    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815978    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:40:22.889540 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815981    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815983    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815985    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815988    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815990    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815993    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815996    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.815998    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816001    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816003    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816006    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816008    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816010    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816013    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816016    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816018    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816023    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816026    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816030    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:40:22.890101 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816032    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816035    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816037    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816040    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816043    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816045    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816047    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816050    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816053    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816055    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816057    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816069    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816073    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816078    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816081    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816457    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816461    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816465    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816467    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816470    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:40:22.890635 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816473    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816475    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816478    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816480    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816483    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816485    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816487    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816490    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816492    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816496    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816499    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816501    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816503    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816505    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816508    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816510    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816513    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816515    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816517    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816519    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:40:22.891186 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816522    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816524    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816527    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816529    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816531    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816537    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816539    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816541    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816544    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816546    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816549    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816551    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816554    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816556    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816559    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816561    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816563    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816566    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816568    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816571    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:40:22.891874 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816573    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816577    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816580    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816582    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816584    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816587    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816589    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816591    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816594    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816596    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816598    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816601    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816603    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816606    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816609    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816611    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816613    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816617    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816619    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:40:22.892721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816621    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816624    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816626    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816628    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816631    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816635    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816639    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816641    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816645    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816647    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816650    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816652    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816655    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816659    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816662    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816664    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816667    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816670    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816674    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816676    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:40:22.893509 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816679    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816681    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816683    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816687    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816690    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816693    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816695    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816697    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816700    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816702    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816707    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816709    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816712    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816714    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.816716    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816789    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816804    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816813    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816819    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816826    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816830    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 10 18:40:22.894023 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816835    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816839    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816843    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816846    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816849    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816853    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816857    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816860    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816862    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816865    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816868    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816871    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816875    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816877    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816880    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816883    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816887    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816893    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816900    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816903    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816906    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816910    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816913    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816916    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 10 18:40:22.894703 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816919    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816921    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816925    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816928    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816931    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816933    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816936    2565 flags.go:64] FLAG: --enable-server="true"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816939    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816945    2565 flags.go:64] FLAG: --event-burst="100"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816948    2565 flags.go:64] FLAG: --event-qps="50"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816950    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816953    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816956    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816961    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816964    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816967    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816970    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816973    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816975    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816978    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816981    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816985    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816987    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816990    2565 flags.go:64] FLAG: --feature-gates=""
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816994    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 10 18:40:22.895410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.816997    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817000    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817003    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817005    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817010    2565 flags.go:64] FLAG: --help="false"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817012    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-132-91.ec2.internal"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817015    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817018    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817021    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817024    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817027    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817030    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817033    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817036    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817038    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817042    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817046    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817049    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817051    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817054    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817058    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817067    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817070    2565 flags.go:64] FLAG: --lock-file=""
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817073    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 10 18:40:22.997902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817075    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 10 18:40:22.901467 ip-10-0-132-91 systemd[1]: Started Kubernetes Kubelet.
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817078    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817084    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817087    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817089    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817092    2565 flags.go:64] FLAG: --logging-format="text"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817095    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817098    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817101    2565 flags.go:64] FLAG: --manifest-url=""
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817104    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817108    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817111    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817116    2565 flags.go:64] FLAG: --max-pods="110"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817119    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817122    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817125    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817128    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817131    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817133    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817141    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817157    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817160    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817162    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 10 18:40:23.015828 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817165    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817169    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817174    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817177    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817180    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817185    2565 flags.go:64] FLAG: --port="10250"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817188    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817191    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-035d39a9442c3053b"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817194    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817197    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817200    2565 flags.go:64] FLAG: --register-node="true"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817203    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817206    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817209    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817212    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817215    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817219    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817222    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817224    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817227    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817230    2565 flags.go:64] FLAG: --runonce="false"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817233    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817237    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817240    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817242    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 10 18:40:23.016807 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817245    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817248    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817251    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817253    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817256    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817259    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817261    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817264    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817267    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817270    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817273    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817278    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817282    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817284    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817288    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817291    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817294    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817296    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817299    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817302    2565 flags.go:64] FLAG: --v="2"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817306    2565 flags.go:64] FLAG: --version="false"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817310    2565 flags.go:64] FLAG: --vmodule=""
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817314    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.817317    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817429    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:40:23.017721 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817433    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817436    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817439    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817442    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817446    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817448    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817451    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817454    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817456    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817459    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817461    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817463    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817466    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817469    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817471    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817474    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817476    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817480    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817482    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817486    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:40:23.018645 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817488    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817491    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817493    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817496    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817498    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817501    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817503    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817507    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817509    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817512    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817514    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817517    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817519    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817521    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817524    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817528    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817532    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817535    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817537    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:40:23.019948 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817540    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817542    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817545    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817547    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817550    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817552    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817555    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817557    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817559    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817562    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817564    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817573    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817577    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817580    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817582    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817585    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817587    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817590    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817592    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817595    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:40:23.020710 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817597    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817600    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817602    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817604    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817607    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817609    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817612    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817614    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817617    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817621    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817623    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817626    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817628    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817631    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817633    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817635    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817638    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817640    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817643    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:40:23.021431 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817645    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817648    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817651    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817654    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817657    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817661    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817664    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817667    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817669    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817672    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817674    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817676    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817679    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817681    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817683    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817686    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817688    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817690    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817693    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:40:23.021987 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.817695    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.818247    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.825491    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.825512    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.826339    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.826348    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826425    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826431    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826436    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826439    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826442    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826445    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826447    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826450    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826454    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:40:23.022611 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826456    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826459    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826461    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826464    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826467    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826470    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826474    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826477    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826479    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826481    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826484    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826487    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826489    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826494    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826496    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826500    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826503    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826506    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826511    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:40:23.023088 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826514    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826517    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826520    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826523    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826525    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826527    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826532    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826534    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826536    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826539    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826541    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826544    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826546    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826550    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826552    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826555    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826557    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826563    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826566    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826569    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:40:23.023691 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826571    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826574    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826578    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826581    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826584    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826587    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826589    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826592    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826595    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826597    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826600    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826603    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826605    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826608    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826611    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826615    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826617    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826620    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826622    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826625    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:40:23.024308 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826627    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826630    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826634    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826636    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826639    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826641    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826643    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826646    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826648    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826653    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826656    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826659    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826661    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826663    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826666    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826670    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826675    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826677    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826680    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826682    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:40:23.024856 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826684    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:40:23.025415 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826689    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:40:23.025415 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826692    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:40:23.025415 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826695    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:40:23.025415 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826697    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:40:23.025415 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826700    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:40:23.025415 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826702    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:40:23.025415 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826705    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:40:23.025415 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826709    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:40:23.025415 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826712    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:40:23.025415 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826714    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:40:23.025415 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.826719    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:40:23.025415 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826852    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:40:23.025415 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826859    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:40:23.025415 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826862    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826865    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826867    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826870    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826872    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826875    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826877    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826879    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826883    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826886    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826890    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826893    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826895    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826897    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826901    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826903    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826908    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826910    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826913    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826915    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:40:23.025839 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826918    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826921    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826923    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826926    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826929    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826931    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826934    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826936    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826938    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826941    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826943    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826947    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826949    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826952    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826954    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826956    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826960    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826963    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826965    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826967    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:40:23.026399 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826971    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826976    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826978    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826982    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826985    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826990    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826994    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826997    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.826999    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827004    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827006    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827009    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827011    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827015    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827018    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827020    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827023    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827027    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827029    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:40:23.026975 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827032    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827035    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827039    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827042    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827044    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827048    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827051    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827053    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827056    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827060    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827067    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827070    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827072    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827077    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827079    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827081    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827084    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827087    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827092    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:40:23.027515 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827096    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827099    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827101    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827106    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827108    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827111    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827113    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827115    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827119    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827122    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827124    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827126    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827128    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827133    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827136    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827142    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827154    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827157    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:40:23.028124 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:22.827160    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:40:23.028627 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.827165    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:40:23.028627 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.827296    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 10 18:40:23.028627 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.829190    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 10 18:40:23.028627 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.830461    2565 server.go:1019] "Starting client certificate rotation"
Jul 10 18:40:23.028627 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.833689    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:40:23.028627 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.855272    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:40:23.028627 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.855339    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:40:23.028627 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.859988    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:40:23.028627 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.866552    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 10 18:40:23.028627 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.868384    2565 log.go:25] "Validated CRI v1 image API"
Jul 10 18:40:23.028627 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.869174    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 10 18:40:23.028627 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.873103    2565 fs.go:135] Filesystem UUIDs: map[3b7134b7-1fe2-4796-966f-f5fb9f707328:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8ca54b83-0b7e-4c07-ac2b-3549fb1ab9f9:/dev/nvme0n1p3]
Jul 10 18:40:23.028944 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.873118    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 10 18:40:23.029038 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.878381    2565 manager.go:216] Machine: {Timestamp:2026-07-10 18:40:22.877233781 +0000 UTC m=+0.345152603 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099986 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e527bf7271a5179ec45d8d8b43547 SystemUUID:ec2e527b-f727-1a51-79ec-45d8d8b43547 BootID:5bfeb62c-6fd6-4155-8ce2-44cd2424212a Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c4:e9:1a:ea:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c4:e9:1a:ea:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:1f:89:e0:86:0d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 10 18:40:23.029038 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.878478    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 10 18:40:23.029038 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.878599    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 10 18:40:23.029038 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.881123    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 10 18:40:23.029038 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.881182    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-91.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 10 18:40:23.029038 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.881832    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 10 18:40:23.029038 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.881844    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 10 18:40:23.029038 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.881851    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.882520    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.883574    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.883889    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.886299    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.886410    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.886437    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.886446    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.886460    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.886471    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.886480    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.886903    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.886928    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.888956    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.891922    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.891960    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.892188    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.892205    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.892213    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.892222    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.892234    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.892243    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.892622    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.892642    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.892653    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.892662    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.892687    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.892699    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.894739    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.894760    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.897493    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-91.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:22.897664    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:22.897663    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.900538    2565 server.go:1268] "Started kubelet"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.900627    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.900613    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.900730    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.901972    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 10 18:40:23.029381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.902326    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:22.905319    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-91.ec2.internal.18c101b953b70e0e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-91.ec2.internal,UID:ip-10-0-132-91.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-91.ec2.internal,},FirstTimestamp:2026-07-10 18:40:22.900510222 +0000 UTC m=+0.368429042,LastTimestamp:2026-07-10 18:40:22.900510222 +0000 UTC m=+0.368429042,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-91.ec2.internal,}"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:22.906035    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.906873    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.906872    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:22.906981    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.907422    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.907429    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.907451    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.907510    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.907519    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.907595    2565 factory.go:55] Registering systemd factory
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.907608    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.907770    2565 factory.go:153] Registering CRI-O factory
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.907783    2565 factory.go:223] Registration of the crio container factory successfully
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.907869    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.907892    2565 factory.go:103] Registering Raw factory
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.907904    2565 manager.go:1196] Started watching for new ooms in manager
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.908211    2565 manager.go:349] Starting recovery of all containers
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.909985    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dnffj"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.915929    2565 manager.go:354] Recovery completed
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.922587    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dnffj"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.927442    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.929354    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.929378    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.929390    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.929430    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.929931    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.929939    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.929955    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.929963    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.930596    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.930616    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.930627    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.931109    2565 policy_none.go:49] "None policy: Start"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.931121    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 10 18:40:23.030392 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.931130    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 10 18:40:23.031544 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.932175    2565 policy_none.go:47] "Start"
Jul 10 18:40:23.031544 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:22.933794    2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-91.ec2.internal\" not found" node="ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.031544 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.961729    2565 manager.go:337] "Starting Device Plugin manager"
Jul 10 18:40:23.031544 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:22.961759    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 10 18:40:23.031544 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.961772    2565 server.go:87] "Starting device plugin registration server"
Jul 10 18:40:23.031544 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.962007    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 10 18:40:23.031544 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.962019    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 10 18:40:23.031544 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.962112    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 10 18:40:23.031544 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.962219    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 10 18:40:23.031544 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:22.962228    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 10 18:40:23.031544 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:22.962704    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 10 18:40:23.031544 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:22.962739    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:23.172787 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.062647    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:40:23.172787 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.064070    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:40:23.172787 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.064103    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:40:23.172787 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.064117    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:40:23.172787 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.064173    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.172787 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.072166    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.172787 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:23.072185    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-91.ec2.internal\": node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:23.172787 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.082444    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 10 18:40:23.172787 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.088355    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 10 18:40:23.172787 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.088390    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 10 18:40:23.172787 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.088412    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 10 18:40:23.172787 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:23.088443    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 10 18:40:23.172787 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.090702    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:40:23.172787 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:23.098654    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.189282    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-91.ec2.internal"]
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.189354    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.190272    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.190297    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.190307    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.191601    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.192206    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.192231    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.192865    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.192884    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.192902    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.193170    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.193185    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.193194    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.194248    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.194269    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.194873    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.194891    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.194901    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:23.198792    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.209457    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/506d760693b7e70a83f18d1881eaae11-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal\" (UID: \"506d760693b7e70a83f18d1881eaae11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.209481    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506d760693b7e70a83f18d1881eaae11-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal\" (UID: \"506d760693b7e70a83f18d1881eaae11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.238785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.209497    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3d2324b5b6efc54e3cd149d6b3cc67c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-91.ec2.internal\" (UID: \"b3d2324b5b6efc54e3cd149d6b3cc67c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.357381 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:23.276973    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-91.ec2.internal\" not found" node="ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.357381 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:23.287095    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-91.ec2.internal\" not found" node="ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.357381 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:23.299107    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:23.357381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.310411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/506d760693b7e70a83f18d1881eaae11-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal\" (UID: \"506d760693b7e70a83f18d1881eaae11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.357381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.310440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506d760693b7e70a83f18d1881eaae11-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal\" (UID: \"506d760693b7e70a83f18d1881eaae11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.357381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.310461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3d2324b5b6efc54e3cd149d6b3cc67c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-91.ec2.internal\" (UID: \"b3d2324b5b6efc54e3cd149d6b3cc67c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.357381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.310507    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3d2324b5b6efc54e3cd149d6b3cc67c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-91.ec2.internal\" (UID: \"b3d2324b5b6efc54e3cd149d6b3cc67c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.357381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.310511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506d760693b7e70a83f18d1881eaae11-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal\" (UID: \"506d760693b7e70a83f18d1881eaae11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.357381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.310510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/506d760693b7e70a83f18d1881eaae11-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal\" (UID: \"506d760693b7e70a83f18d1881eaae11\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.496769 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:23.400069    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:23.503243 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:23.500588    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:23.603939 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:23.600910    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:23.695779 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.648240    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.695779 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.648242    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:23.783296 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:23.701493    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:23.801781 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:23.801751    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:23.856121 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.856095    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 10 18:40:23.856255 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.856239    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:40:23.856312 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.856271    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:40:23.901950 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:23.901923    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:23.925909 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.925879    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 18:35:22 +0000 UTC" deadline="2028-01-19 12:53:38.223839012 +0000 UTC"
Jul 10 18:40:23.925909 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:23.925903    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13386h13m14.297938373s"
Jul 10 18:40:24.002191 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:24.002132    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:24.102646 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:24.102565    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:24.202971 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:24.202936    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:24.303343 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:24.303315    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:24.365272 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:24.365235    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506d760693b7e70a83f18d1881eaae11.slice/crio-ec08b799f9d0e79049028124e73c7bd21d909ec032569fa3866fff60a132144c WatchSource:0}: Error finding container ec08b799f9d0e79049028124e73c7bd21d909ec032569fa3866fff60a132144c: Status 404 returned error can't find the container with id ec08b799f9d0e79049028124e73c7bd21d909ec032569fa3866fff60a132144c
Jul 10 18:40:24.369731 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.369714    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:40:24.403416 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:24.403387    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-91.ec2.internal\" not found"
Jul 10 18:40:24.413085 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.413066    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:40:24.445568 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:24.445542    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d2324b5b6efc54e3cd149d6b3cc67c.slice/crio-7fafb29e33e42afe76bdf90368ebf057b827b94587d57725ba3af9462d4fface WatchSource:0}: Error finding container 7fafb29e33e42afe76bdf90368ebf057b827b94587d57725ba3af9462d4fface: Status 404 returned error can't find the container with id 7fafb29e33e42afe76bdf90368ebf057b827b94587d57725ba3af9462d4fface
Jul 10 18:40:24.479655 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.479624    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:40:24.507506 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.507486    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:24.519418 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.519400    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:40:24.520376 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.520365    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-91.ec2.internal"
Jul 10 18:40:24.528740 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.528728    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:40:24.856343 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.856317    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:40:24.887971 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.887945    2565 apiserver.go:52] "Watching apiserver"
Jul 10 18:40:24.900437 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.900410    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 10 18:40:24.901780 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.901756    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-87nb6","kube-system/global-pull-secret-syncer-sljz9","kube-system/konnectivity-agent-nkk6q","kube-system/kube-apiserver-proxy-ip-10-0-132-91.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9","openshift-cluster-node-tuning-operator/tuned-qc7h5","openshift-dns/node-resolver-vb9ww","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal","openshift-multus/multus-additional-cni-plugins-xvx67","openshift-image-registry/node-ca-bzjdq","openshift-multus/multus-jfz8j","openshift-multus/network-metrics-daemon-dj7xt","openshift-network-diagnostics/network-check-target-crm4n","openshift-network-operator/iptables-alerter-8bxmt"]
Jul 10 18:40:24.903848 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.903606    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bzjdq"
Jul 10 18:40:24.906049 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.905449    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.906797 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.906777    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 10 18:40:24.906992 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.906974    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 10 18:40:24.908055 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.908030    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 10 18:40:24.908319 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.908302    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 10 18:40:24.908486 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.908473    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 10 18:40:24.908800 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.908785    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 10 18:40:24.909002 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.908985    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ksgkx\""
Jul 10 18:40:24.909202 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.909187    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5jrmj\""
Jul 10 18:40:24.909433 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.909371    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 10 18:40:24.909570 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.909553    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 10 18:40:24.909783 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.909768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nkk6q"
Jul 10 18:40:24.910643 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.910627    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8bxmt"
Jul 10 18:40:24.911612 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.911595    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vb9ww"
Jul 10 18:40:24.913203 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.912083    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.913203 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.912485    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k62x8\""
Jul 10 18:40:24.913203 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.912911    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 10 18:40:24.913203 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.912996    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 10 18:40:24.913203 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.913178    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 10 18:40:24.916171 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.915513    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 10 18:40:24.916171 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.915540    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:24.916171 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.916097    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 10 18:40:24.916346 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.916212    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gfc7j\""
Jul 10 18:40:24.916346 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.916248    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 10 18:40:24.916346 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.916336    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 10 18:40:24.916483 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.916397    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 10 18:40:24.916483 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.916411    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 10 18:40:24.916569 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.916514    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9rffz\""
Jul 10 18:40:24.916569 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.916561    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 10 18:40:24.916657 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.916570    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:40:24.916852 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.916834    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 10 18:40:24.916918 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.916852    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 10 18:40:24.916984 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.916971    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-frq6l\""
Jul 10 18:40:24.917690 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917385    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29xfm\" (UniqueName: \"kubernetes.io/projected/975a4c98-91a8-4277-b3ff-a5cd2da229cb-kube-api-access-29xfm\") pod \"node-ca-bzjdq\" (UID: \"975a4c98-91a8-4277-b3ff-a5cd2da229cb\") " pod="openshift-image-registry/node-ca-bzjdq"
Jul 10 18:40:24.917690 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-run-multus-certs\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.917690 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917443    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-kubelet\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.917690 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917463    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-cni-bin\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.917690 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.917690 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqqlz\" (UniqueName: \"kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz\") pod \"network-check-target-crm4n\" (UID: \"e265d96b-328a-4f5d-aceb-ecd5a32693c1\") " pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:24.917690 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917528    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-system-cni-dir\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.917690 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-os-release\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.917690 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917596    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-var-lib-openvswitch\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.917690 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be7bd63d-2e6e-4570-abf3-272865bfef5e-os-release\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:24.917690 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917641    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be7bd63d-2e6e-4570-abf3-272865bfef5e-cni-binary-copy\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:24.917690 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917678    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-hostroot\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.917690 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917698    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/638a939b-46d2-4d09-a1a8-ee95f22d51d1-host-slash\") pod \"iptables-alerter-8bxmt\" (UID: \"638a939b-46d2-4d09-a1a8-ee95f22d51d1\") " pod="openshift-network-operator/iptables-alerter-8bxmt"
Jul 10 18:40:24.918309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917720    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4cffd840-0b56-4aaf-8c26-32c36bff4d51-env-overrides\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.918309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917759    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/be7bd63d-2e6e-4570-abf3-272865bfef5e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:24.918309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917782    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-multus-conf-dir\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.918309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917803    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/638a939b-46d2-4d09-a1a8-ee95f22d51d1-iptables-alerter-script\") pod \"iptables-alerter-8bxmt\" (UID: \"638a939b-46d2-4d09-a1a8-ee95f22d51d1\") " pod="openshift-network-operator/iptables-alerter-8bxmt"
Jul 10 18:40:24.918309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917825    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-log-socket\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.918309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917845    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-multus-cni-dir\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.918309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917866    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f165c26-834e-4a90-b142-8293f6860d4f-cni-binary-copy\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.918309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917901    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4cffd840-0b56-4aaf-8c26-32c36bff4d51-ovn-node-metrics-cert\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.918309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqvf2\" (UniqueName: \"kubernetes.io/projected/4cffd840-0b56-4aaf-8c26-32c36bff4d51-kube-api-access-cqvf2\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.918309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-run-netns\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.918309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d46n\" (UniqueName: \"kubernetes.io/projected/638a939b-46d2-4d09-a1a8-ee95f22d51d1-kube-api-access-7d46n\") pod \"iptables-alerter-8bxmt\" (UID: \"638a939b-46d2-4d09-a1a8-ee95f22d51d1\") " pod="openshift-network-operator/iptables-alerter-8bxmt"
Jul 10 18:40:24.918309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.917998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/975a4c98-91a8-4277-b3ff-a5cd2da229cb-host\") pod \"node-ca-bzjdq\" (UID: \"975a4c98-91a8-4277-b3ff-a5cd2da229cb\") " pod="openshift-image-registry/node-ca-bzjdq"
Jul 10 18:40:24.918309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918018    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-var-lib-cni-bin\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.918309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918039    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f165c26-834e-4a90-b142-8293f6860d4f-multus-daemon-config\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.918897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918069    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-node-log\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.918897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918112    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-cni-netd\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.918897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs572\" (UniqueName: \"kubernetes.io/projected/be7bd63d-2e6e-4570-abf3-272865bfef5e-kube-api-access-rs572\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:24.918897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918190    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-multus-socket-dir-parent\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.918897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/344d8617-d6ba-4abe-a743-a638e7f0bfad-konnectivity-ca\") pod \"konnectivity-agent-nkk6q\" (UID: \"344d8617-d6ba-4abe-a743-a638e7f0bfad\") " pod="kube-system/konnectivity-agent-nkk6q"
Jul 10 18:40:24.918897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918243    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be7bd63d-2e6e-4570-abf3-272865bfef5e-system-cni-dir\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:24.918897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918265    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-run-k8s-cni-cncf-io\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.918897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918306    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-var-lib-kubelet\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.918897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918327    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48f95840-5a3c-4014-b1c5-f489153c94ce-hosts-file\") pod \"node-resolver-vb9ww\" (UID: \"48f95840-5a3c-4014-b1c5-f489153c94ce\") " pod="openshift-dns/node-resolver-vb9ww"
Jul 10 18:40:24.918897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be7bd63d-2e6e-4570-abf3-272865bfef5e-cnibin\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:24.918897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918401    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/344d8617-d6ba-4abe-a743-a638e7f0bfad-agent-certs\") pod \"konnectivity-agent-nkk6q\" (UID: \"344d8617-d6ba-4abe-a743-a638e7f0bfad\") " pod="kube-system/konnectivity-agent-nkk6q"
Jul 10 18:40:24.918897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918425    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg52l\" (UniqueName: \"kubernetes.io/projected/48f95840-5a3c-4014-b1c5-f489153c94ce-kube-api-access-jg52l\") pod \"node-resolver-vb9ww\" (UID: \"48f95840-5a3c-4014-b1c5-f489153c94ce\") " pod="openshift-dns/node-resolver-vb9ww"
Jul 10 18:40:24.918897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-systemd-units\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.918897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918486    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-run-netns\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.919549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-run-systemd\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.919549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918529    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f95840-5a3c-4014-b1c5-f489153c94ce-tmp-dir\") pod \"node-resolver-vb9ww\" (UID: \"48f95840-5a3c-4014-b1c5-f489153c94ce\") " pod="openshift-dns/node-resolver-vb9ww"
Jul 10 18:40:24.919549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/be7bd63d-2e6e-4570-abf3-272865bfef5e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:24.919549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-run-ovn\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.919549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsbm4\" (UniqueName: \"kubernetes.io/projected/8f165c26-834e-4a90-b142-8293f6860d4f-kube-api-access-hsbm4\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.919549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918635    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-etc-openvswitch\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.919549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918715    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-run-ovn-kubernetes\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.919549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918742    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4cffd840-0b56-4aaf-8c26-32c36bff4d51-ovnkube-script-lib\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.919549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918781    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-var-lib-cni-multus\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.919549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918807    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be7bd63d-2e6e-4570-abf3-272865bfef5e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:24.919549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/975a4c98-91a8-4277-b3ff-a5cd2da229cb-serviceca\") pod \"node-ca-bzjdq\" (UID: \"975a4c98-91a8-4277-b3ff-a5cd2da229cb\") " pod="openshift-image-registry/node-ca-bzjdq"
Jul 10 18:40:24.919549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918857    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-cnibin\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.919549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-etc-kubernetes\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:24.919549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4cffd840-0b56-4aaf-8c26-32c36bff4d51-ovnkube-config\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.920171 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.918988    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-slash\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.920171 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.919027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-run-openvswitch\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:24.920171 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.919436    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:24.920171 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:24.919491    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:24.920171 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.919799    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 10 18:40:24.920171 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.919983    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 10 18:40:24.920425 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.920270    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6xsv7\""
Jul 10 18:40:24.921180 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.921070    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:24.921180 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:24.921138    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:24.922546 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.922330    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:24.924117 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.924100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:24.932124 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.932102    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 10 18:40:24.932231 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.932131    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:40:24.932379 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.932362    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 10 18:40:24.932560 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.932537    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:40:24.932623 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.932567    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 10 18:40:24.932790 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.932771    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9swvg\""
Jul 10 18:40:24.933007 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.932990    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vwjw7\""
Jul 10 18:40:24.933074 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.933042    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 10 18:40:24.933949 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.933926    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:24.934032 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:24.933988    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:24.946744 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.946726    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dctg7"
Jul 10 18:40:24.954184 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:24.954136    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dctg7"
Jul 10 18:40:25.009160 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.009121    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 10 18:40:25.019846 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.019822    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqlz\" (UniqueName: \"kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz\") pod \"network-check-target-crm4n\" (UID: \"e265d96b-328a-4f5d-aceb-ecd5a32693c1\") " pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:25.019980 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.019856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-system-cni-dir\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.019980 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.019880    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-os-release\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.019980 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.019899    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-var-lib-openvswitch\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.019980 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.019922    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-systemd\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.019980 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.019964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-tmp\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.020253 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.019991    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-os-release\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.020253 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020026    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncg6q\" (UniqueName: \"kubernetes.io/projected/f6a55653-cedb-4274-8cbc-badf0c84c2c8-kube-api-access-ncg6q\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.020253 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020048    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs\") pod \"network-metrics-daemon-dj7xt\" (UID: \"b90039f7-1ab3-4de0-81fe-4829c384ed89\") " pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:25.020253 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be7bd63d-2e6e-4570-abf3-272865bfef5e-os-release\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.020253 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020096    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-system-cni-dir\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.020253 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020110    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-var-lib-openvswitch\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.020253 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be7bd63d-2e6e-4570-abf3-272865bfef5e-cni-binary-copy\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.020253 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be7bd63d-2e6e-4570-abf3-272865bfef5e-os-release\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.020253 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020211    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-hostroot\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.020253 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/638a939b-46d2-4d09-a1a8-ee95f22d51d1-host-slash\") pod \"iptables-alerter-8bxmt\" (UID: \"638a939b-46d2-4d09-a1a8-ee95f22d51d1\") " pod="openshift-network-operator/iptables-alerter-8bxmt"
Jul 10 18:40:25.020685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4cffd840-0b56-4aaf-8c26-32c36bff4d51-env-overrides\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.020685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020284    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-sys\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.020685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/be7bd63d-2e6e-4570-abf3-272865bfef5e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.020685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-multus-conf-dir\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.020685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/638a939b-46d2-4d09-a1a8-ee95f22d51d1-iptables-alerter-script\") pod \"iptables-alerter-8bxmt\" (UID: \"638a939b-46d2-4d09-a1a8-ee95f22d51d1\") " pod="openshift-network-operator/iptables-alerter-8bxmt"
Jul 10 18:40:25.020685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-log-socket\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.020685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020406    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-sysconfig\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.020685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-host\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.020685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020449    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-multus-cni-dir\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.020685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f165c26-834e-4a90-b142-8293f6860d4f-cni-binary-copy\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.020685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020493    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4cffd840-0b56-4aaf-8c26-32c36bff4d51-ovn-node-metrics-cert\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.020685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqvf2\" (UniqueName: \"kubernetes.io/projected/4cffd840-0b56-4aaf-8c26-32c36bff4d51-kube-api-access-cqvf2\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.020685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020540    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-sys-fs\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.020685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020559    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-multus-conf-dir\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.021307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020563    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-run-netns\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.021307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-run-netns\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.021307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d46n\" (UniqueName: \"kubernetes.io/projected/638a939b-46d2-4d09-a1a8-ee95f22d51d1-kube-api-access-7d46n\") pod \"iptables-alerter-8bxmt\" (UID: \"638a939b-46d2-4d09-a1a8-ee95f22d51d1\") " pod="openshift-network-operator/iptables-alerter-8bxmt"
Jul 10 18:40:25.021307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020640    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-sysctl-d\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.021307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-device-dir\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.021307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020688    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/975a4c98-91a8-4277-b3ff-a5cd2da229cb-host\") pod \"node-ca-bzjdq\" (UID: \"975a4c98-91a8-4277-b3ff-a5cd2da229cb\") " pod="openshift-image-registry/node-ca-bzjdq"
Jul 10 18:40:25.021307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020718    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-var-lib-cni-bin\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.021307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f165c26-834e-4a90-b142-8293f6860d4f-multus-daemon-config\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.021307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020762    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-node-log\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.021307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020799    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-cni-netd\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.021307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020822    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-sysctl-conf\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.021307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-socket-dir\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.021307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfdpd\" (UniqueName: \"kubernetes.io/projected/b90039f7-1ab3-4de0-81fe-4829c384ed89-kube-api-access-tfdpd\") pod \"network-metrics-daemon-dj7xt\" (UID: \"b90039f7-1ab3-4de0-81fe-4829c384ed89\") " pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:25.021307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs572\" (UniqueName: \"kubernetes.io/projected/be7bd63d-2e6e-4570-abf3-272865bfef5e-kube-api-access-rs572\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.021741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-multus-socket-dir-parent\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.021741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020950    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/344d8617-d6ba-4abe-a743-a638e7f0bfad-konnectivity-ca\") pod \"konnectivity-agent-nkk6q\" (UID: \"344d8617-d6ba-4abe-a743-a638e7f0bfad\") " pod="kube-system/konnectivity-agent-nkk6q"
Jul 10 18:40:25.021741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.020973    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-lib-modules\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.021741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021011    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be7bd63d-2e6e-4570-abf3-272865bfef5e-system-cni-dir\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.021741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021033    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-run-k8s-cni-cncf-io\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.021741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-var-lib-kubelet\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.021741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-log-socket\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.021741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48f95840-5a3c-4014-b1c5-f489153c94ce-hosts-file\") pod \"node-resolver-vb9ww\" (UID: \"48f95840-5a3c-4014-b1c5-f489153c94ce\") " pod="openshift-dns/node-resolver-vb9ww"
Jul 10 18:40:25.021741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:25.021741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021167    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.021741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021212    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be7bd63d-2e6e-4570-abf3-272865bfef5e-cnibin\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.021741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/344d8617-d6ba-4abe-a743-a638e7f0bfad-agent-certs\") pod \"konnectivity-agent-nkk6q\" (UID: \"344d8617-d6ba-4abe-a743-a638e7f0bfad\") " pod="kube-system/konnectivity-agent-nkk6q"
Jul 10 18:40:25.021741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg52l\" (UniqueName: \"kubernetes.io/projected/48f95840-5a3c-4014-b1c5-f489153c94ce-kube-api-access-jg52l\") pod \"node-resolver-vb9ww\" (UID: \"48f95840-5a3c-4014-b1c5-f489153c94ce\") " pod="openshift-dns/node-resolver-vb9ww"
Jul 10 18:40:25.021741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021284    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be7bd63d-2e6e-4570-abf3-272865bfef5e-cni-binary-copy\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021310    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f165c26-834e-4a90-b142-8293f6860d4f-cni-binary-copy\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021385    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-multus-cni-dir\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021446    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/975a4c98-91a8-4277-b3ff-a5cd2da229cb-host\") pod \"node-ca-bzjdq\" (UID: \"975a4c98-91a8-4277-b3ff-a5cd2da229cb\") " pod="openshift-image-registry/node-ca-bzjdq"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-hostroot\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxm4w\" (UniqueName: \"kubernetes.io/projected/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-kube-api-access-nxm4w\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021869    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021131    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/638a939b-46d2-4d09-a1a8-ee95f22d51d1-host-slash\") pod \"iptables-alerter-8bxmt\" (UID: \"638a939b-46d2-4d09-a1a8-ee95f22d51d1\") " pod="openshift-network-operator/iptables-alerter-8bxmt"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-systemd-units\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021941    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48f95840-5a3c-4014-b1c5-f489153c94ce-hosts-file\") pod \"node-resolver-vb9ww\" (UID: \"48f95840-5a3c-4014-b1c5-f489153c94ce\") " pod="openshift-dns/node-resolver-vb9ww"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021945    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-run-netns\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-run-systemd\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021984    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-run-netns\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021952    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-multus-socket-dir-parent\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022021    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be7bd63d-2e6e-4570-abf3-272865bfef5e-cnibin\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.022112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-run-systemd\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.022565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.021685    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-run-k8s-cni-cncf-io\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.022565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022053    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-var-lib-kubelet\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.022565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f95840-5a3c-4014-b1c5-f489153c94ce-tmp-dir\") pod \"node-resolver-vb9ww\" (UID: \"48f95840-5a3c-4014-b1c5-f489153c94ce\") " pod="openshift-dns/node-resolver-vb9ww"
Jul 10 18:40:25.022565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022105    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be7bd63d-2e6e-4570-abf3-272865bfef5e-system-cni-dir\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.022565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-kubernetes\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.022565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-tuned\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.022565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022161    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/be7bd63d-2e6e-4570-abf3-272865bfef5e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.022565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022179    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/be7bd63d-2e6e-4570-abf3-272865bfef5e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.022565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022204    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-run-ovn\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.022565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022226    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-var-lib-kubelet\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.022565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsbm4\" (UniqueName: \"kubernetes.io/projected/8f165c26-834e-4a90-b142-8293f6860d4f-kube-api-access-hsbm4\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.022565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022273    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-etc-openvswitch\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.022565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-run-ovn-kubernetes\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.022565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4cffd840-0b56-4aaf-8c26-32c36bff4d51-ovnkube-script-lib\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6730ada-8f92-421c-8768-422891b4c4f9-dbus\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-registration-dir\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-var-lib-cni-multus\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f95840-5a3c-4014-b1c5-f489153c94ce-tmp-dir\") pod \"node-resolver-vb9ww\" (UID: \"48f95840-5a3c-4014-b1c5-f489153c94ce\") " pod="openshift-dns/node-resolver-vb9ww"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be7bd63d-2e6e-4570-abf3-272865bfef5e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022439    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/975a4c98-91a8-4277-b3ff-a5cd2da229cb-serviceca\") pod \"node-ca-bzjdq\" (UID: \"975a4c98-91a8-4277-b3ff-a5cd2da229cb\") " pod="openshift-image-registry/node-ca-bzjdq"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022486    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-systemd-units\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022489    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022553    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4cffd840-0b56-4aaf-8c26-32c36bff4d51-env-overrides\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-var-lib-cni-bin\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022643    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-node-log\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022676    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-etc-openvswitch\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022695    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-run-ovn-kubernetes\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022795    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/638a939b-46d2-4d09-a1a8-ee95f22d51d1-iptables-alerter-script\") pod \"iptables-alerter-8bxmt\" (UID: \"638a939b-46d2-4d09-a1a8-ee95f22d51d1\") " pod="openshift-network-operator/iptables-alerter-8bxmt"
Jul 10 18:40:25.023192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/975a4c98-91a8-4277-b3ff-a5cd2da229cb-serviceca\") pod \"node-ca-bzjdq\" (UID: \"975a4c98-91a8-4277-b3ff-a5cd2da229cb\") " pod="openshift-image-registry/node-ca-bzjdq"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-cnibin\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022865    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-cni-netd\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022881    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6730ada-8f92-421c-8768-422891b4c4f9-kubelet-config\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-run\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-var-lib-cni-multus\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022980    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f165c26-834e-4a90-b142-8293f6860d4f-multus-daemon-config\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.022972    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-etc-kubernetes\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023031    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-cnibin\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-etc-kubernetes\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023075    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-run-ovn\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023104    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4cffd840-0b56-4aaf-8c26-32c36bff4d51-ovnkube-config\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-modprobe-d\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023132    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be7bd63d-2e6e-4570-abf3-272865bfef5e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-slash\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.024444 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023195    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-run-openvswitch\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.025089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29xfm\" (UniqueName: \"kubernetes.io/projected/975a4c98-91a8-4277-b3ff-a5cd2da229cb-kube-api-access-29xfm\") pod \"node-ca-bzjdq\" (UID: \"975a4c98-91a8-4277-b3ff-a5cd2da229cb\") " pod="openshift-image-registry/node-ca-bzjdq"
Jul 10 18:40:25.025089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-run-multus-certs\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.025089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-kubelet\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.025089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-cni-bin\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.025089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-run-openvswitch\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.025089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.025089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-cni-bin\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.025089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023379    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.025089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023402    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-slash\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.025089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023405    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f165c26-834e-4a90-b142-8293f6860d4f-host-run-multus-certs\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.025089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023433    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cffd840-0b56-4aaf-8c26-32c36bff4d51-host-kubelet\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.025089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023562    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4cffd840-0b56-4aaf-8c26-32c36bff4d51-ovnkube-config\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.025089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4cffd840-0b56-4aaf-8c26-32c36bff4d51-ovnkube-script-lib\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.025089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.023817    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/be7bd63d-2e6e-4570-abf3-272865bfef5e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.025702 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.024350    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/344d8617-d6ba-4abe-a743-a638e7f0bfad-konnectivity-ca\") pod \"konnectivity-agent-nkk6q\" (UID: \"344d8617-d6ba-4abe-a743-a638e7f0bfad\") " pod="kube-system/konnectivity-agent-nkk6q"
Jul 10 18:40:25.025750 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.025703    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:25.025750 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.025724    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:25.025750 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.025737    2565 projected.go:196] Error preparing data for projected volume kube-api-access-qqqlz for pod openshift-network-diagnostics/network-check-target-crm4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:25.025925 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.025798    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz podName:e265d96b-328a-4f5d-aceb-ecd5a32693c1 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:25.525779763 +0000 UTC m=+2.993698585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qqqlz" (UniqueName: "kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz") pod "network-check-target-crm4n" (UID: "e265d96b-328a-4f5d-aceb-ecd5a32693c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:25.026083 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.026054    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4cffd840-0b56-4aaf-8c26-32c36bff4d51-ovn-node-metrics-cert\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.026955 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.026934    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/344d8617-d6ba-4abe-a743-a638e7f0bfad-agent-certs\") pod \"konnectivity-agent-nkk6q\" (UID: \"344d8617-d6ba-4abe-a743-a638e7f0bfad\") " pod="kube-system/konnectivity-agent-nkk6q"
Jul 10 18:40:25.032124 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.031347    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d46n\" (UniqueName: \"kubernetes.io/projected/638a939b-46d2-4d09-a1a8-ee95f22d51d1-kube-api-access-7d46n\") pod \"iptables-alerter-8bxmt\" (UID: \"638a939b-46d2-4d09-a1a8-ee95f22d51d1\") " pod="openshift-network-operator/iptables-alerter-8bxmt"
Jul 10 18:40:25.033930 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.033873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29xfm\" (UniqueName: \"kubernetes.io/projected/975a4c98-91a8-4277-b3ff-a5cd2da229cb-kube-api-access-29xfm\") pod \"node-ca-bzjdq\" (UID: \"975a4c98-91a8-4277-b3ff-a5cd2da229cb\") " pod="openshift-image-registry/node-ca-bzjdq"
Jul 10 18:40:25.034019 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.033946    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsbm4\" (UniqueName: \"kubernetes.io/projected/8f165c26-834e-4a90-b142-8293f6860d4f-kube-api-access-hsbm4\") pod \"multus-jfz8j\" (UID: \"8f165c26-834e-4a90-b142-8293f6860d4f\") " pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.034861 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.034824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg52l\" (UniqueName: \"kubernetes.io/projected/48f95840-5a3c-4014-b1c5-f489153c94ce-kube-api-access-jg52l\") pod \"node-resolver-vb9ww\" (UID: \"48f95840-5a3c-4014-b1c5-f489153c94ce\") " pod="openshift-dns/node-resolver-vb9ww"
Jul 10 18:40:25.034952 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.034932    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs572\" (UniqueName: \"kubernetes.io/projected/be7bd63d-2e6e-4570-abf3-272865bfef5e-kube-api-access-rs572\") pod \"multus-additional-cni-plugins-xvx67\" (UID: \"be7bd63d-2e6e-4570-abf3-272865bfef5e\") " pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.037646 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.037626    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqvf2\" (UniqueName: \"kubernetes.io/projected/4cffd840-0b56-4aaf-8c26-32c36bff4d51-kube-api-access-cqvf2\") pod \"ovnkube-node-87nb6\" (UID: \"4cffd840-0b56-4aaf-8c26-32c36bff4d51\") " pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.071810 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.071779    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:40:25.094726 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.094673    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-91.ec2.internal" event={"ID":"b3d2324b5b6efc54e3cd149d6b3cc67c","Type":"ContainerStarted","Data":"7fafb29e33e42afe76bdf90368ebf057b827b94587d57725ba3af9462d4fface"}
Jul 10 18:40:25.097513 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.097452    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal" event={"ID":"506d760693b7e70a83f18d1881eaae11","Type":"ContainerStarted","Data":"ec08b799f9d0e79049028124e73c7bd21d909ec032569fa3866fff60a132144c"}
Jul 10 18:40:25.124637 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124560    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxm4w\" (UniqueName: \"kubernetes.io/projected/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-kube-api-access-nxm4w\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.124637 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124601    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.124637 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-kubernetes\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.124886 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-tuned\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.124886 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124674    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-var-lib-kubelet\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.124886 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6730ada-8f92-421c-8768-422891b4c4f9-dbus\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:25.124886 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124719    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-registration-dir\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.124886 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6730ada-8f92-421c-8768-422891b4c4f9-kubelet-config\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:25.124886 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-run\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.124886 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-modprobe-d\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.124886 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124835    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-systemd\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.124886 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124845    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-var-lib-kubelet\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.124886 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-tmp\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.124886 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124849    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-run\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.124886 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124881    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncg6q\" (UniqueName: \"kubernetes.io/projected/f6a55653-cedb-4274-8cbc-badf0c84c2c8-kube-api-access-ncg6q\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.124886 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-registration-dir\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.125480 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124895    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6730ada-8f92-421c-8768-422891b4c4f9-kubelet-config\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:25.125480 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs\") pod \"network-metrics-daemon-dj7xt\" (UID: \"b90039f7-1ab3-4de0-81fe-4829c384ed89\") " pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:25.125480 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124907    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6730ada-8f92-421c-8768-422891b4c4f9-dbus\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:25.125480 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124852    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.125480 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-sys\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.125480 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-sysconfig\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.125480 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.124998    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-host\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.125480 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.125006    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:25.125480 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-sys-fs\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.125480 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-sysctl-d\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.125480 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125046    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-systemd\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.125480 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125091    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-sys\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.125480 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.125096    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs podName:b90039f7-1ab3-4de0-81fe-4829c384ed89 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:25.625071194 +0000 UTC m=+3.092990007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs") pod "network-metrics-daemon-dj7xt" (UID: "b90039f7-1ab3-4de0-81fe-4829c384ed89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:25.125480 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-sysctl-d\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.126102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125165    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-sysconfig\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.126102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-device-dir\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.126102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125202    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-host\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.126102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-sysctl-conf\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.126102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-sys-fs\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.126102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-socket-dir\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.126102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125103    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-kubernetes\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.126102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125004    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-modprobe-d\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.126102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfdpd\" (UniqueName: \"kubernetes.io/projected/b90039f7-1ab3-4de0-81fe-4829c384ed89-kube-api-access-tfdpd\") pod \"network-metrics-daemon-dj7xt\" (UID: \"b90039f7-1ab3-4de0-81fe-4829c384ed89\") " pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:25.126102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-lib-modules\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.126102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125323    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:25.126102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125346    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.126102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-socket-dir\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.126102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125404    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-sysctl-conf\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.126697 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125408    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.126697 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125295    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6a55653-cedb-4274-8cbc-badf0c84c2c8-device-dir\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.126697 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.125474    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:25.126697 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.125473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-lib-modules\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.126697 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.125540    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret podName:d6730ada-8f92-421c-8768-422891b4c4f9 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:25.625525407 +0000 UTC m=+3.093444219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret") pod "global-pull-secret-syncer-sljz9" (UID: "d6730ada-8f92-421c-8768-422891b4c4f9") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:25.127332 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.127290    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-etc-tuned\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.128613 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.128589    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-tmp\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.138664 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.138611    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfdpd\" (UniqueName: \"kubernetes.io/projected/b90039f7-1ab3-4de0-81fe-4829c384ed89-kube-api-access-tfdpd\") pod \"network-metrics-daemon-dj7xt\" (UID: \"b90039f7-1ab3-4de0-81fe-4829c384ed89\") " pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:25.140105 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.140079    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxm4w\" (UniqueName: \"kubernetes.io/projected/2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b-kube-api-access-nxm4w\") pod \"tuned-qc7h5\" (UID: \"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b\") " pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.140207 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.140098    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncg6q\" (UniqueName: \"kubernetes.io/projected/f6a55653-cedb-4274-8cbc-badf0c84c2c8-kube-api-access-ncg6q\") pod \"aws-ebs-csi-driver-node-64pr9\" (UID: \"f6a55653-cedb-4274-8cbc-badf0c84c2c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.233411 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.233222    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bzjdq"
Jul 10 18:40:25.243357 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.243328    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jfz8j"
Jul 10 18:40:25.253196 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.253170    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nkk6q"
Jul 10 18:40:25.259971 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.259815    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8bxmt"
Jul 10 18:40:25.268356 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.268334    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vb9ww"
Jul 10 18:40:25.274049 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.274031    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:25.281554 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.281537    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xvx67"
Jul 10 18:40:25.289034 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.289018    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qc7h5"
Jul 10 18:40:25.295492 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.295476    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9"
Jul 10 18:40:25.528882 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.528803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqlz\" (UniqueName: \"kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz\") pod \"network-check-target-crm4n\" (UID: \"e265d96b-328a-4f5d-aceb-ecd5a32693c1\") " pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:25.529028 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.528947    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:25.529028 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.528963    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:25.529028 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.528972    2565 projected.go:196] Error preparing data for projected volume kube-api-access-qqqlz for pod openshift-network-diagnostics/network-check-target-crm4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:25.529204 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.529028    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz podName:e265d96b-328a-4f5d-aceb-ecd5a32693c1 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:26.529009489 +0000 UTC m=+3.996928315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqqlz" (UniqueName: "kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz") pod "network-check-target-crm4n" (UID: "e265d96b-328a-4f5d-aceb-ecd5a32693c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:25.629367 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.629340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs\") pod \"network-metrics-daemon-dj7xt\" (UID: \"b90039f7-1ab3-4de0-81fe-4829c384ed89\") " pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:25.629517 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.629383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:25.629517 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.629471    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:25.629517 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.629472    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:25.629644 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.629524    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret podName:d6730ada-8f92-421c-8768-422891b4c4f9 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:26.629509198 +0000 UTC m=+4.097428009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret") pod "global-pull-secret-syncer-sljz9" (UID: "d6730ada-8f92-421c-8768-422891b4c4f9") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:25.629644 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:25.629536    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs podName:b90039f7-1ab3-4de0-81fe-4829c384ed89 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:26.629531027 +0000 UTC m=+4.097449835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs") pod "network-metrics-daemon-dj7xt" (UID: "b90039f7-1ab3-4de0-81fe-4829c384ed89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:25.956258 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.956176    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:35:24 +0000 UTC" deadline="2028-02-03 17:45:53.28597013 +0000 UTC"
Jul 10 18:40:25.956258 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:25.956207    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13751h5m27.329766082s"
Jul 10 18:40:26.063842 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:26.063807    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975a4c98_91a8_4277_b3ff_a5cd2da229cb.slice/crio-3849fbe515a3362adbb20b2aeb32e54a787932e63e299c87c496df51a4c19daa WatchSource:0}: Error finding container 3849fbe515a3362adbb20b2aeb32e54a787932e63e299c87c496df51a4c19daa: Status 404 returned error can't find the container with id 3849fbe515a3362adbb20b2aeb32e54a787932e63e299c87c496df51a4c19daa
Jul 10 18:40:26.065995 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:26.065955    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344d8617_d6ba_4abe_a743_a638e7f0bfad.slice/crio-64bd06c6148a1db01d971b285999ec4c4b1c40a6b0eeabd5f5f44fa8c91c392e WatchSource:0}: Error finding container 64bd06c6148a1db01d971b285999ec4c4b1c40a6b0eeabd5f5f44fa8c91c392e: Status 404 returned error can't find the container with id 64bd06c6148a1db01d971b285999ec4c4b1c40a6b0eeabd5f5f44fa8c91c392e
Jul 10 18:40:26.069376 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:26.069356    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638a939b_46d2_4d09_a1a8_ee95f22d51d1.slice/crio-5e5283eac7bdc8b1113e0ff39354b651f774441e26de9e2ad3cda8e350ee4e00 WatchSource:0}: Error finding container 5e5283eac7bdc8b1113e0ff39354b651f774441e26de9e2ad3cda8e350ee4e00: Status 404 returned error can't find the container with id 5e5283eac7bdc8b1113e0ff39354b651f774441e26de9e2ad3cda8e350ee4e00
Jul 10 18:40:26.070290 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:26.070197    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cffd840_0b56_4aaf_8c26_32c36bff4d51.slice/crio-afcfb907005a02d7a4c6ebbd96448f5ac9b72ae710ee22b46a6637e01852281e WatchSource:0}: Error finding container afcfb907005a02d7a4c6ebbd96448f5ac9b72ae710ee22b46a6637e01852281e: Status 404 returned error can't find the container with id afcfb907005a02d7a4c6ebbd96448f5ac9b72ae710ee22b46a6637e01852281e
Jul 10 18:40:26.071086 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:26.071058    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f95840_5a3c_4014_b1c5_f489153c94ce.slice/crio-f2cde2f680b3c71ff80e9966d14899255ab15e3e3682a0a6b675c3c2a7a0d615 WatchSource:0}: Error finding container f2cde2f680b3c71ff80e9966d14899255ab15e3e3682a0a6b675c3c2a7a0d615: Status 404 returned error can't find the container with id f2cde2f680b3c71ff80e9966d14899255ab15e3e3682a0a6b675c3c2a7a0d615
Jul 10 18:40:26.071809 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:26.071708    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a55653_cedb_4274_8cbc_badf0c84c2c8.slice/crio-3eb7541fef6291e4e152d7890c312cc36b91692aeb88bfca8c4abe528ede620c WatchSource:0}: Error finding container 3eb7541fef6291e4e152d7890c312cc36b91692aeb88bfca8c4abe528ede620c: Status 404 returned error can't find the container with id 3eb7541fef6291e4e152d7890c312cc36b91692aeb88bfca8c4abe528ede620c
Jul 10 18:40:26.073439 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:26.073282    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a4e7d_7bd8_4c03_b8e6_de54f623ed3b.slice/crio-001d5e9444da54d5113ae6420d24131944e56b3ebd0564610464cb796dcf701c WatchSource:0}: Error finding container 001d5e9444da54d5113ae6420d24131944e56b3ebd0564610464cb796dcf701c: Status 404 returned error can't find the container with id 001d5e9444da54d5113ae6420d24131944e56b3ebd0564610464cb796dcf701c
Jul 10 18:40:26.074191 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:26.074170    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7bd63d_2e6e_4570_abf3_272865bfef5e.slice/crio-fb87113dee6d2bab8058756b099be218c36af2c9829a3e65b2b5c10e3d6ba184 WatchSource:0}: Error finding container fb87113dee6d2bab8058756b099be218c36af2c9829a3e65b2b5c10e3d6ba184: Status 404 returned error can't find the container with id fb87113dee6d2bab8058756b099be218c36af2c9829a3e65b2b5c10e3d6ba184
Jul 10 18:40:26.089164 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.089120    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:26.089270 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:26.089235    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:26.103585 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.103559    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvx67" event={"ID":"be7bd63d-2e6e-4570-abf3-272865bfef5e","Type":"ContainerStarted","Data":"fb87113dee6d2bab8058756b099be218c36af2c9829a3e65b2b5c10e3d6ba184"}
Jul 10 18:40:26.109345 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.109316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qc7h5" event={"ID":"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b","Type":"ContainerStarted","Data":"001d5e9444da54d5113ae6420d24131944e56b3ebd0564610464cb796dcf701c"}
Jul 10 18:40:26.110463 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.110440    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6" event={"ID":"4cffd840-0b56-4aaf-8c26-32c36bff4d51","Type":"ContainerStarted","Data":"afcfb907005a02d7a4c6ebbd96448f5ac9b72ae710ee22b46a6637e01852281e"}
Jul 10 18:40:26.111505 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.111483    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bzjdq" event={"ID":"975a4c98-91a8-4277-b3ff-a5cd2da229cb","Type":"ContainerStarted","Data":"3849fbe515a3362adbb20b2aeb32e54a787932e63e299c87c496df51a4c19daa"}
Jul 10 18:40:26.112544 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.112522    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jfz8j" event={"ID":"8f165c26-834e-4a90-b142-8293f6860d4f","Type":"ContainerStarted","Data":"b0837a7002e47c652c152b9e56f2f78388db65f9047d8380b2493f01205b32bd"}
Jul 10 18:40:26.113642 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.113487    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9" event={"ID":"f6a55653-cedb-4274-8cbc-badf0c84c2c8","Type":"ContainerStarted","Data":"3eb7541fef6291e4e152d7890c312cc36b91692aeb88bfca8c4abe528ede620c"}
Jul 10 18:40:26.115943 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.115923    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vb9ww" event={"ID":"48f95840-5a3c-4014-b1c5-f489153c94ce","Type":"ContainerStarted","Data":"f2cde2f680b3c71ff80e9966d14899255ab15e3e3682a0a6b675c3c2a7a0d615"}
Jul 10 18:40:26.119284 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.119259    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8bxmt" event={"ID":"638a939b-46d2-4d09-a1a8-ee95f22d51d1","Type":"ContainerStarted","Data":"5e5283eac7bdc8b1113e0ff39354b651f774441e26de9e2ad3cda8e350ee4e00"}
Jul 10 18:40:26.120916 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.120896    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nkk6q" event={"ID":"344d8617-d6ba-4abe-a743-a638e7f0bfad","Type":"ContainerStarted","Data":"64bd06c6148a1db01d971b285999ec4c4b1c40a6b0eeabd5f5f44fa8c91c392e"}
Jul 10 18:40:26.536498 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.536461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqlz\" (UniqueName: \"kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz\") pod \"network-check-target-crm4n\" (UID: \"e265d96b-328a-4f5d-aceb-ecd5a32693c1\") " pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:26.536695 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:26.536674    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:26.536762 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:26.536702    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:26.536762 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:26.536715    2565 projected.go:196] Error preparing data for projected volume kube-api-access-qqqlz for pod openshift-network-diagnostics/network-check-target-crm4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:26.536854 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:26.536771    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz podName:e265d96b-328a-4f5d-aceb-ecd5a32693c1 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:28.536754192 +0000 UTC m=+6.004673005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqqlz" (UniqueName: "kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz") pod "network-check-target-crm4n" (UID: "e265d96b-328a-4f5d-aceb-ecd5a32693c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:26.638559 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.637684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:26.638559 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.637765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs\") pod \"network-metrics-daemon-dj7xt\" (UID: \"b90039f7-1ab3-4de0-81fe-4829c384ed89\") " pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:26.638559 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:26.637888    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:26.638559 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:26.637947    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs podName:b90039f7-1ab3-4de0-81fe-4829c384ed89 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:28.637927832 +0000 UTC m=+6.105846644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs") pod "network-metrics-daemon-dj7xt" (UID: "b90039f7-1ab3-4de0-81fe-4829c384ed89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:26.638559 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:26.638321    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:26.638559 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:26.638370    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret podName:d6730ada-8f92-421c-8768-422891b4c4f9 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:28.638356112 +0000 UTC m=+6.106274934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret") pod "global-pull-secret-syncer-sljz9" (UID: "d6730ada-8f92-421c-8768-422891b4c4f9") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:26.956809 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.956611    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:35:24 +0000 UTC" deadline="2028-01-15 13:50:43.717633174 +0000 UTC"
Jul 10 18:40:26.956809 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:26.956671    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13291h10m16.760966859s"
Jul 10 18:40:27.092745 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:27.092712    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:27.092923 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:27.092836    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:27.093443 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:27.093239    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:27.093443 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:27.093348    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:27.140041 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:27.139857    2565 generic.go:356] "Generic (PLEG): container finished" podID="506d760693b7e70a83f18d1881eaae11" containerID="c54a932ad11a00c04a35fc2256c0a9c054f4c0292efe6ec119584b3a4989482d" exitCode=0
Jul 10 18:40:27.140041 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:27.139972    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal" event={"ID":"506d760693b7e70a83f18d1881eaae11","Type":"ContainerDied","Data":"c54a932ad11a00c04a35fc2256c0a9c054f4c0292efe6ec119584b3a4989482d"}
Jul 10 18:40:27.144358 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:27.144328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-91.ec2.internal" event={"ID":"b3d2324b5b6efc54e3cd149d6b3cc67c","Type":"ContainerStarted","Data":"f2ac11e719dd2e1a202872cad9eb3a03fe91edaf01d895d628ce79b3e86d17f2"}
Jul 10 18:40:28.089704 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:28.089542    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:28.089704 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:28.089700    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:28.166340 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:28.166262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal" event={"ID":"506d760693b7e70a83f18d1881eaae11","Type":"ContainerStarted","Data":"0ed2f0e320ab88ca6a35ded1918087ebb91bc760b605230ff872e742ae5cd514"}
Jul 10 18:40:28.183332 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:28.183269    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-91.ec2.internal" podStartSLOduration=4.183138836 podStartE2EDuration="4.183138836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:27.173258453 +0000 UTC m=+4.641177287" watchObservedRunningTime="2026-07-10 18:40:28.183138836 +0000 UTC m=+5.651057671"
Jul 10 18:40:28.555093 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:28.554427    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqlz\" (UniqueName: \"kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz\") pod \"network-check-target-crm4n\" (UID: \"e265d96b-328a-4f5d-aceb-ecd5a32693c1\") " pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:28.555093 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:28.554610    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:28.555093 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:28.554628    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:28.555093 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:28.554640    2565 projected.go:196] Error preparing data for projected volume kube-api-access-qqqlz for pod openshift-network-diagnostics/network-check-target-crm4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:28.555093 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:28.554696    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz podName:e265d96b-328a-4f5d-aceb-ecd5a32693c1 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:32.554677277 +0000 UTC m=+10.022596092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqqlz" (UniqueName: "kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz") pod "network-check-target-crm4n" (UID: "e265d96b-328a-4f5d-aceb-ecd5a32693c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:28.656137 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:28.655414    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs\") pod \"network-metrics-daemon-dj7xt\" (UID: \"b90039f7-1ab3-4de0-81fe-4829c384ed89\") " pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:28.656137 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:28.655483    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:28.656137 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:28.655604    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:28.656137 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:28.655663    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret podName:d6730ada-8f92-421c-8768-422891b4c4f9 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:32.655644335 +0000 UTC m=+10.123563147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret") pod "global-pull-secret-syncer-sljz9" (UID: "d6730ada-8f92-421c-8768-422891b4c4f9") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:28.656137 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:28.656054    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:28.656137 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:28.656102    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs podName:b90039f7-1ab3-4de0-81fe-4829c384ed89 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:32.656087341 +0000 UTC m=+10.124006156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs") pod "network-metrics-daemon-dj7xt" (UID: "b90039f7-1ab3-4de0-81fe-4829c384ed89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:29.088653 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:29.088616    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:29.088830 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:29.088765    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:29.089213 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:29.089191    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:29.089331 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:29.089307    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:30.089555 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:30.089518    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:30.090049 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:30.089664    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:31.093249 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:31.093175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:31.093682 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:31.093300    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:31.093682 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:31.093392    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:31.093682 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:31.093484    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:32.088738 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:32.088698    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:32.088923 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:32.088843    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:32.592556 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:32.592517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqlz\" (UniqueName: \"kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz\") pod \"network-check-target-crm4n\" (UID: \"e265d96b-328a-4f5d-aceb-ecd5a32693c1\") " pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:32.593022 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:32.592770    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:32.593022 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:32.592814    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:32.593022 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:32.592828    2565 projected.go:196] Error preparing data for projected volume kube-api-access-qqqlz for pod openshift-network-diagnostics/network-check-target-crm4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:32.593022 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:32.592900    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz podName:e265d96b-328a-4f5d-aceb-ecd5a32693c1 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:40.592880167 +0000 UTC m=+18.060798980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqqlz" (UniqueName: "kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz") pod "network-check-target-crm4n" (UID: "e265d96b-328a-4f5d-aceb-ecd5a32693c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:32.694410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:32.693836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs\") pod \"network-metrics-daemon-dj7xt\" (UID: \"b90039f7-1ab3-4de0-81fe-4829c384ed89\") " pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:32.694410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:32.693913    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:32.694410 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:32.694046    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:32.694410 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:32.694109    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret podName:d6730ada-8f92-421c-8768-422891b4c4f9 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:40.694089017 +0000 UTC m=+18.162007830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret") pod "global-pull-secret-syncer-sljz9" (UID: "d6730ada-8f92-421c-8768-422891b4c4f9") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:32.694410 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:32.694263    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:32.694410 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:32.694332    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs podName:b90039f7-1ab3-4de0-81fe-4829c384ed89 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:40.694313355 +0000 UTC m=+18.162232185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs") pod "network-metrics-daemon-dj7xt" (UID: "b90039f7-1ab3-4de0-81fe-4829c384ed89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:33.092559 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:33.092477    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:33.092723 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:33.092595    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:33.092934 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:33.092915    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:33.093047 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:33.093028    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:34.089989 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:34.089237    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:34.089989 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:34.089435    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:35.089415 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:35.089366    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:35.089613 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:35.089366    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:35.089613 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:35.089507    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:35.089613 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:35.089600    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:36.089269 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:36.089226    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:36.089690 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:36.089353    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:37.088924 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:37.088868    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:37.088924 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:37.088904    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:37.089204 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:37.089019    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:37.089204 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:37.089109    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:38.089517 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:38.089485    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:38.089962 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:38.089609    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:39.089253 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:39.089221    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:39.089850 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:39.089220    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:39.090283 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:39.089908    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:39.092593 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:39.090762    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:40.088942 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:40.088903    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:40.089194 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:40.089036    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:40.652464 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:40.652200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqlz\" (UniqueName: \"kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz\") pod \"network-check-target-crm4n\" (UID: \"e265d96b-328a-4f5d-aceb-ecd5a32693c1\") " pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:40.652942 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:40.652369    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:40.652942 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:40.652549    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:40.652942 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:40.652564    2565 projected.go:196] Error preparing data for projected volume kube-api-access-qqqlz for pod openshift-network-diagnostics/network-check-target-crm4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:40.652942 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:40.652631    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz podName:e265d96b-328a-4f5d-aceb-ecd5a32693c1 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:56.652612534 +0000 UTC m=+34.120531352 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqqlz" (UniqueName: "kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz") pod "network-check-target-crm4n" (UID: "e265d96b-328a-4f5d-aceb-ecd5a32693c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:40.753858 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:40.753811    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:40.754028 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:40.753979    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:40.754028 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:40.754007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs\") pod \"network-metrics-daemon-dj7xt\" (UID: \"b90039f7-1ab3-4de0-81fe-4829c384ed89\") " pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:40.754230 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:40.754043    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret podName:d6730ada-8f92-421c-8768-422891b4c4f9 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:56.754024195 +0000 UTC m=+34.221943012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret") pod "global-pull-secret-syncer-sljz9" (UID: "d6730ada-8f92-421c-8768-422891b4c4f9") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:40.754230 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:40.754100    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:40.754230 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:40.754188    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs podName:b90039f7-1ab3-4de0-81fe-4829c384ed89 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:56.754138092 +0000 UTC m=+34.222056906 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs") pod "network-metrics-daemon-dj7xt" (UID: "b90039f7-1ab3-4de0-81fe-4829c384ed89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:41.089591 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:41.089504    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:41.089763 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:41.089640    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:41.089763 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:41.089709    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:41.089863 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:41.089835    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:42.089249 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:42.089217    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:42.089722 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:42.089339    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:43.089691 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:43.089651    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:43.090138 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:43.089803    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:43.090138 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:43.089809    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:43.090138 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:43.089897    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:44.089210 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:44.089168    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:44.089373 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:44.089307    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:45.089380 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.089115    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:45.089380 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:45.089318    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:45.090429 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.089526    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:45.090429 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:45.089732    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:45.203218 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.202959    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jfz8j" event={"ID":"8f165c26-834e-4a90-b142-8293f6860d4f","Type":"ContainerStarted","Data":"db3b11a6ff0a31cade34083b1aea51016e288c06cf67b6011f5e15ebd4359799"}
Jul 10 18:40:45.204513 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.204480    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9" event={"ID":"f6a55653-cedb-4274-8cbc-badf0c84c2c8","Type":"ContainerStarted","Data":"c7e102f6f96fcc3f9d06cd2b4ddc2979b4318f7e3e1a5b5c68197db1c41e0191"}
Jul 10 18:40:45.205692 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.205667    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vb9ww" event={"ID":"48f95840-5a3c-4014-b1c5-f489153c94ce","Type":"ContainerStarted","Data":"09da85da597d6783b62352aba17b7a80d01ca52eb4cb87d2d130ee89adeb2af8"}
Jul 10 18:40:45.207098 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.207076    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nkk6q" event={"ID":"344d8617-d6ba-4abe-a743-a638e7f0bfad","Type":"ContainerStarted","Data":"ad7f9749550e5b415fd7ee4ff22aa64cedfbfd8acb8aade79f9e2e2700cb71e2"}
Jul 10 18:40:45.208810 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.208784    2565 generic.go:356] "Generic (PLEG): container finished" podID="be7bd63d-2e6e-4570-abf3-272865bfef5e" containerID="9bdc3fa618ecb0997e3e544dcfd9fdd1f8a68bc7403b8d791a59f283853d7870" exitCode=0
Jul 10 18:40:45.208924 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.208868    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvx67" event={"ID":"be7bd63d-2e6e-4570-abf3-272865bfef5e","Type":"ContainerDied","Data":"9bdc3fa618ecb0997e3e544dcfd9fdd1f8a68bc7403b8d791a59f283853d7870"}
Jul 10 18:40:45.210411 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.210381    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qc7h5" event={"ID":"2a0a4e7d-7bd8-4c03-b8e6-de54f623ed3b","Type":"ContainerStarted","Data":"8ccb02af2624806d99f5cca7e9c01d20dbccb1febaa3f6017b96cf376135a20d"}
Jul 10 18:40:45.212194 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.212174    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6" event={"ID":"4cffd840-0b56-4aaf-8c26-32c36bff4d51","Type":"ContainerStarted","Data":"2239f8a8ae8aefa51f27d206336fbd97ea1b2b2e029233aad8742e37b6bc26c8"}
Jul 10 18:40:45.212352 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.212228    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6" event={"ID":"4cffd840-0b56-4aaf-8c26-32c36bff4d51","Type":"ContainerStarted","Data":"eccc597f4c5ce96ce2ef36d79cb56712d23cd9bcb011415fe573970786645f37"}
Jul 10 18:40:45.213658 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.213640    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bzjdq" event={"ID":"975a4c98-91a8-4277-b3ff-a5cd2da229cb","Type":"ContainerStarted","Data":"66b67cca054d1d27fcc4b6cd1cfa1c0e0a441e606b359f6c4161bc22919f6f82"}
Jul 10 18:40:45.229536 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.229461    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-91.ec2.internal" podStartSLOduration=21.229436491 podStartE2EDuration="21.229436491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:28.183810357 +0000 UTC m=+5.651729207" watchObservedRunningTime="2026-07-10 18:40:45.229436491 +0000 UTC m=+22.697355361"
Jul 10 18:40:45.229789 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.229754    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jfz8j" podStartSLOduration=4.376152063 podStartE2EDuration="23.229745565s" totalImagesPullingTime="18.853593502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:45.229437483 +0000 UTC m=+22.697356317" watchObservedRunningTime="2026-07-10 18:40:45.229745565 +0000 UTC m=+22.697664401"
Jul 10 18:40:45.249737 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.249674    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nkk6q" podStartSLOduration=9.066103028 podStartE2EDuration="23.249651001s" totalImagesPullingTime="14.183547973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:45.248413506 +0000 UTC m=+22.716332338" watchObservedRunningTime="2026-07-10 18:40:45.249651001 +0000 UTC m=+22.717569835"
Jul 10 18:40:45.271681 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.271629    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vb9ww" podStartSLOduration=4.469759178 podStartE2EDuration="23.271611623s" totalImagesPullingTime="18.801852445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:45.271137688 +0000 UTC m=+22.739056520" watchObservedRunningTime="2026-07-10 18:40:45.271611623 +0000 UTC m=+22.739530455"
Jul 10 18:40:45.318895 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.318847    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bzjdq" podStartSLOduration=4.50734042 podStartE2EDuration="23.318831578s" totalImagesPullingTime="18.811491158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:45.318561733 +0000 UTC m=+22.786480566" watchObservedRunningTime="2026-07-10 18:40:45.318831578 +0000 UTC m=+22.786750407"
Jul 10 18:40:45.340899 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:45.340857    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qc7h5" podStartSLOduration=3.539471562 podStartE2EDuration="22.340839785s" totalImagesPullingTime="18.801368223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:45.340709495 +0000 UTC m=+22.808628350" watchObservedRunningTime="2026-07-10 18:40:45.340839785 +0000 UTC m=+22.808758616"
Jul 10 18:40:46.088649 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:46.088614    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:46.088828 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:46.088743    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:46.220132 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:46.220092    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6" event={"ID":"4cffd840-0b56-4aaf-8c26-32c36bff4d51","Type":"ContainerStarted","Data":"1f8893875d08fd975251a10bc5205168015e3dfc0260c3821c3ea0ed88faa234"}
Jul 10 18:40:46.220679 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:46.220209    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6" event={"ID":"4cffd840-0b56-4aaf-8c26-32c36bff4d51","Type":"ContainerStarted","Data":"a96f1e2f8aafd72d86d470f128039f4ea09ef993e37442416aa3fb34f1642543"}
Jul 10 18:40:46.220679 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:46.220262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6" event={"ID":"4cffd840-0b56-4aaf-8c26-32c36bff4d51","Type":"ContainerStarted","Data":"ea3fd3752e5e21a086c2e21414147cb257448b5a79267ae5c6ebdd8f7bfa246d"}
Jul 10 18:40:46.220679 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:46.220311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6" event={"ID":"4cffd840-0b56-4aaf-8c26-32c36bff4d51","Type":"ContainerStarted","Data":"3cb87da96d082fb6e5fb34c0785210ce65fcdb1e1e70d39a855781196008ee05"}
Jul 10 18:40:46.515375 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:46.515114    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 10 18:40:46.937521 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:46.937482    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nkk6q"
Jul 10 18:40:46.938217 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:46.938192    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nkk6q"
Jul 10 18:40:46.976645 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:46.976524    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T18:40:46.51536653Z","UUID":"df74baf7-ae34-4602-a893-62cf32f23273","Handler":null,"Name":"","Endpoint":""}
Jul 10 18:40:46.978390 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:46.978367    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 10 18:40:46.978528 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:46.978411    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 10 18:40:47.089088 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:47.089057    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:47.089286 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:47.089217    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:47.089504 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:47.089057    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:47.089504 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:47.089471    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:47.224432 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:47.224329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8bxmt" event={"ID":"638a939b-46d2-4d09-a1a8-ee95f22d51d1","Type":"ContainerStarted","Data":"a1e4e01631d3059b2a77be192b41885df1b4eb5bc5d2152eaad35b2e91810da4"}
Jul 10 18:40:47.226325 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:47.226287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9" event={"ID":"f6a55653-cedb-4274-8cbc-badf0c84c2c8","Type":"ContainerStarted","Data":"d828657ee483fe051decd49fab0497884d3d271cbe8b1aef0294fc2f80aaa80d"}
Jul 10 18:40:47.226585 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:47.226528    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nkk6q"
Jul 10 18:40:47.227063 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:47.227046    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nkk6q"
Jul 10 18:40:47.268504 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:47.268444    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8bxmt" podStartSLOduration=6.465242633 podStartE2EDuration="25.268426052s" totalImagesPullingTime="18.803183419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:47.247816534 +0000 UTC m=+24.715735366" watchObservedRunningTime="2026-07-10 18:40:47.268426052 +0000 UTC m=+24.736344883"
Jul 10 18:40:48.089261 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:48.089220    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:48.089433 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:48.089365    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:48.232747 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:48.232703    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6" event={"ID":"4cffd840-0b56-4aaf-8c26-32c36bff4d51","Type":"ContainerStarted","Data":"92f7b181bf4e8a2011238a14d084cf211ac18dda79b19f8ca2732f78b25f6f40"}
Jul 10 18:40:48.235020 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:48.234980    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9" event={"ID":"f6a55653-cedb-4274-8cbc-badf0c84c2c8","Type":"ContainerStarted","Data":"1a1489286d77c990215226fb2f72e69dac8f9bd378e3fccc9db82eb52d0c9b6c"}
Jul 10 18:40:49.089554 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:49.089512    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:49.089842 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:49.089653    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:49.089842 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:49.089706    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:49.089979 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:49.089840    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:50.089304 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:50.089090    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:50.090045 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:50.089397    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:50.241824 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:50.241711    2565 generic.go:356] "Generic (PLEG): container finished" podID="be7bd63d-2e6e-4570-abf3-272865bfef5e" containerID="df19a526cf471a3df1f9f04b8827346d573d8fe3ab1aec0e3ff3ed21d1ebd23d" exitCode=0
Jul 10 18:40:50.241824 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:50.241758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvx67" event={"ID":"be7bd63d-2e6e-4570-abf3-272865bfef5e","Type":"ContainerDied","Data":"df19a526cf471a3df1f9f04b8827346d573d8fe3ab1aec0e3ff3ed21d1ebd23d"}
Jul 10 18:40:50.245430 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:50.245407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6" event={"ID":"4cffd840-0b56-4aaf-8c26-32c36bff4d51","Type":"ContainerStarted","Data":"13561fd771a01cc121b3d0e2b8aa8af8bd1d6354349c15aa81e17eecef0fd265"}
Jul 10 18:40:50.245712 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:50.245692    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:50.245873 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:50.245751    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:50.245873 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:50.245783    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:50.262041 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:50.262017    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:50.262209 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:50.262108    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:40:50.266733 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:50.266694    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64pr9" podStartSLOduration=6.111547423 podStartE2EDuration="27.266682471s" totalImagesPullingTime="21.155135048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:48.259273973 +0000 UTC m=+25.727192817" watchObservedRunningTime="2026-07-10 18:40:50.266682471 +0000 UTC m=+27.734601303"
Jul 10 18:40:51.089475 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:51.089389    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:51.089973 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:51.089385    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:51.089973 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:51.089516    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:51.089973 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:51.089616    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:52.089634 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:52.089454    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:52.089999 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:52.089713    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:52.093479 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:52.093431    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6" podStartSLOduration=11.228668686 podStartE2EDuration="30.09341061s" totalImagesPullingTime="18.864741924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:50.294241963 +0000 UTC m=+27.762160816" watchObservedRunningTime="2026-07-10 18:40:52.09341061 +0000 UTC m=+29.561329443"
Jul 10 18:40:52.094105 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:52.094079    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sljz9"]
Jul 10 18:40:52.097026 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:52.097007    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-crm4n"]
Jul 10 18:40:52.097122 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:52.097081    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:52.097197 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:52.097183    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:52.100033 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:52.100015    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dj7xt"]
Jul 10 18:40:52.100135 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:52.100093    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:52.100223 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:52.100186    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:52.252257 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:52.252166    2565 generic.go:356] "Generic (PLEG): container finished" podID="be7bd63d-2e6e-4570-abf3-272865bfef5e" containerID="9742f1d72b6c9ecbc700f9110faa12afd2c65f39ed840781fa725bbb5a49c333" exitCode=0
Jul 10 18:40:52.252257 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:52.252233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvx67" event={"ID":"be7bd63d-2e6e-4570-abf3-272865bfef5e","Type":"ContainerDied","Data":"9742f1d72b6c9ecbc700f9110faa12afd2c65f39ed840781fa725bbb5a49c333"}
Jul 10 18:40:52.252257 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:52.252255    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:52.252477 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:52.252448    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:54.088764 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:54.088726    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:54.089410 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:54.088832    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:54.089410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:54.088733    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:54.089410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:54.088864    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:54.089410 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:54.088946    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:54.089410 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:54.089013    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:54.259195 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:54.259073    2565 generic.go:356] "Generic (PLEG): container finished" podID="be7bd63d-2e6e-4570-abf3-272865bfef5e" containerID="34d315eba7a2971e1df3724dae69a83b7c95b0eee081423ddf3d86250156a90b" exitCode=0
Jul 10 18:40:54.259195 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:54.259128    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvx67" event={"ID":"be7bd63d-2e6e-4570-abf3-272865bfef5e","Type":"ContainerDied","Data":"34d315eba7a2971e1df3724dae69a83b7c95b0eee081423ddf3d86250156a90b"}
Jul 10 18:40:56.088981 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:56.088943    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:56.089654 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:56.089000    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:56.089654 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:56.088952    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:56.089654 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:56.089064    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:56.089654 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:56.089181    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:56.089654 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:56.089272    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:56.674992 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:56.674960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqlz\" (UniqueName: \"kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz\") pod \"network-check-target-crm4n\" (UID: \"e265d96b-328a-4f5d-aceb-ecd5a32693c1\") " pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:56.675223 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:56.675170    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:56.675223 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:56.675197    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:56.675223 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:56.675212    2565 projected.go:196] Error preparing data for projected volume kube-api-access-qqqlz for pod openshift-network-diagnostics/network-check-target-crm4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:56.675446 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:56.675281    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz podName:e265d96b-328a-4f5d-aceb-ecd5a32693c1 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:28.675260708 +0000 UTC m=+66.143179532 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqqlz" (UniqueName: "kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz") pod "network-check-target-crm4n" (UID: "e265d96b-328a-4f5d-aceb-ecd5a32693c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:56.775734 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:56.775692    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs\") pod \"network-metrics-daemon-dj7xt\" (UID: \"b90039f7-1ab3-4de0-81fe-4829c384ed89\") " pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:56.775924 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:56.775767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:56.775924 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:56.775877    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:56.775924 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:56.775898    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:56.776081 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:56.775958    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs podName:b90039f7-1ab3-4de0-81fe-4829c384ed89 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:28.775937832 +0000 UTC m=+66.243856655 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs") pod "network-metrics-daemon-dj7xt" (UID: "b90039f7-1ab3-4de0-81fe-4829c384ed89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:56.776081 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:56.775982    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret podName:d6730ada-8f92-421c-8768-422891b4c4f9 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:28.775971627 +0000 UTC m=+66.243890451 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret") pod "global-pull-secret-syncer-sljz9" (UID: "d6730ada-8f92-421c-8768-422891b4c4f9") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:58.088938 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.088904    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:40:58.088938 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.088930    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:40:58.088938 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.088945    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:40:58.089466 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:58.089039    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dj7xt" podUID="b90039f7-1ab3-4de0-81fe-4829c384ed89"
Jul 10 18:40:58.089466 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:58.089172    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sljz9" podUID="d6730ada-8f92-421c-8768-422891b4c4f9"
Jul 10 18:40:58.089466 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:58.089230    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-crm4n" podUID="e265d96b-328a-4f5d-aceb-ecd5a32693c1"
Jul 10 18:40:58.811935 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.811905    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-91.ec2.internal" event="NodeReady"
Jul 10 18:40:58.812095 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.812035    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 10 18:40:58.867869 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.867836    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gp9v9"]
Jul 10 18:40:58.893548 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.893519    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8zfrf"]
Jul 10 18:40:58.893717 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.893662    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:40:58.896337 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.896120    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 10 18:40:58.896337 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.896260    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 10 18:40:58.896515 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.896355    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4x2b4\""
Jul 10 18:40:58.907671 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.907650    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gp9v9"]
Jul 10 18:40:58.907774 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.907683    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8zfrf"]
Jul 10 18:40:58.907774 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.907757    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8zfrf"
Jul 10 18:40:58.910381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.910364    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 10 18:40:58.910948 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.910932    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 10 18:40:58.911025 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.910936    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 10 18:40:58.911083 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.911050    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g2pth\""
Jul 10 18:40:58.992634 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.992598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsq5p\" (UniqueName: \"kubernetes.io/projected/52c473ba-b012-460e-89a1-e32d7b7dc013-kube-api-access-gsq5p\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:40:58.992634 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.992639    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52c473ba-b012-460e-89a1-e32d7b7dc013-tmp-dir\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:40:58.992852 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.992671    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t2zc\" (UniqueName: \"kubernetes.io/projected/6d6e03d6-117c-4854-b76c-dba72a1a2f27-kube-api-access-4t2zc\") pod \"ingress-canary-8zfrf\" (UID: \"6d6e03d6-117c-4854-b76c-dba72a1a2f27\") " pod="openshift-ingress-canary/ingress-canary-8zfrf"
Jul 10 18:40:58.992852 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.992711    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:40:58.992852 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.992739    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52c473ba-b012-460e-89a1-e32d7b7dc013-config-volume\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:40:58.992852 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:58.992773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d6e03d6-117c-4854-b76c-dba72a1a2f27-cert\") pod \"ingress-canary-8zfrf\" (UID: \"6d6e03d6-117c-4854-b76c-dba72a1a2f27\") " pod="openshift-ingress-canary/ingress-canary-8zfrf"
Jul 10 18:40:59.093597 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.093571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsq5p\" (UniqueName: \"kubernetes.io/projected/52c473ba-b012-460e-89a1-e32d7b7dc013-kube-api-access-gsq5p\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:40:59.093968 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.093604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52c473ba-b012-460e-89a1-e32d7b7dc013-tmp-dir\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:40:59.093968 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.093729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t2zc\" (UniqueName: \"kubernetes.io/projected/6d6e03d6-117c-4854-b76c-dba72a1a2f27-kube-api-access-4t2zc\") pod \"ingress-canary-8zfrf\" (UID: \"6d6e03d6-117c-4854-b76c-dba72a1a2f27\") " pod="openshift-ingress-canary/ingress-canary-8zfrf"
Jul 10 18:40:59.093968 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.093786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:40:59.093968 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.093826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52c473ba-b012-460e-89a1-e32d7b7dc013-config-volume\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:40:59.093968 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.093855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d6e03d6-117c-4854-b76c-dba72a1a2f27-cert\") pod \"ingress-canary-8zfrf\" (UID: \"6d6e03d6-117c-4854-b76c-dba72a1a2f27\") " pod="openshift-ingress-canary/ingress-canary-8zfrf"
Jul 10 18:40:59.093968 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.093882    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52c473ba-b012-460e-89a1-e32d7b7dc013-tmp-dir\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:40:59.093968 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:59.093952    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:59.094350 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:59.094032    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls podName:52c473ba-b012-460e-89a1-e32d7b7dc013 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:59.594011098 +0000 UTC m=+37.061929922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls") pod "dns-default-gp9v9" (UID: "52c473ba-b012-460e-89a1-e32d7b7dc013") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:59.094414 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.094394    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52c473ba-b012-460e-89a1-e32d7b7dc013-config-volume\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:40:59.098249 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.098230    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d6e03d6-117c-4854-b76c-dba72a1a2f27-cert\") pod \"ingress-canary-8zfrf\" (UID: \"6d6e03d6-117c-4854-b76c-dba72a1a2f27\") " pod="openshift-ingress-canary/ingress-canary-8zfrf"
Jul 10 18:40:59.102367 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.102337    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsq5p\" (UniqueName: \"kubernetes.io/projected/52c473ba-b012-460e-89a1-e32d7b7dc013-kube-api-access-gsq5p\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:40:59.102840 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.102824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t2zc\" (UniqueName: \"kubernetes.io/projected/6d6e03d6-117c-4854-b76c-dba72a1a2f27-kube-api-access-4t2zc\") pod \"ingress-canary-8zfrf\" (UID: \"6d6e03d6-117c-4854-b76c-dba72a1a2f27\") " pod="openshift-ingress-canary/ingress-canary-8zfrf"
Jul 10 18:40:59.216179 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.215985    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8zfrf"
Jul 10 18:40:59.276445 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.276412    2565 generic.go:356] "Generic (PLEG): container finished" podID="be7bd63d-2e6e-4570-abf3-272865bfef5e" containerID="c92953250523e5e540d83ef45cd9055a1d12e0e5c6308ce1c64594b97dd8df04" exitCode=0
Jul 10 18:40:59.276584 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.276456    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvx67" event={"ID":"be7bd63d-2e6e-4570-abf3-272865bfef5e","Type":"ContainerDied","Data":"c92953250523e5e540d83ef45cd9055a1d12e0e5c6308ce1c64594b97dd8df04"}
Jul 10 18:40:59.412354 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.412327    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8zfrf"]
Jul 10 18:40:59.439260 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:40:59.439228    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6e03d6_117c_4854_b76c_dba72a1a2f27.slice/crio-75411b59f32830bdc5ebdbf508d4e25be8960db7cc1150972a80efaca7fcdd51 WatchSource:0}: Error finding container 75411b59f32830bdc5ebdbf508d4e25be8960db7cc1150972a80efaca7fcdd51: Status 404 returned error can't find the container with id 75411b59f32830bdc5ebdbf508d4e25be8960db7cc1150972a80efaca7fcdd51
Jul 10 18:40:59.440210 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.440194    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f4w65"]
Jul 10 18:40:59.442195 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.442178    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:40:59.447967 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.447949    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 10 18:40:59.448069 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.448056    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 10 18:40:59.448379 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.448353    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nfnkx\""
Jul 10 18:40:59.448544 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.448525    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 10 18:40:59.448919 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.448860    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 10 18:40:59.505729 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.505699    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f4w65"]
Jul 10 18:40:59.597515 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.597425    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f43cba3-15f4-4170-8b22-36f666bffe9d-crio-socket\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:40:59.597515 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.597457    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxk7j\" (UniqueName: \"kubernetes.io/projected/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-api-access-rxk7j\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:40:59.597515 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.597502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:40:59.597746 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:59.597583    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:59.597746 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:59.597631    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls podName:52c473ba-b012-460e-89a1-e32d7b7dc013 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:00.597618021 +0000 UTC m=+38.065536831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls") pod "dns-default-gp9v9" (UID: "52c473ba-b012-460e-89a1-e32d7b7dc013") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:59.597746 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.597693    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:40:59.597746 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.597732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f43cba3-15f4-4170-8b22-36f666bffe9d-data-volume\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:40:59.597903 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.597769    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:40:59.698739 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.698700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f43cba3-15f4-4170-8b22-36f666bffe9d-data-volume\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:40:59.698739 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.698743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:40:59.698988 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:59.698816    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:40:59.698988 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:59.698871    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm podName:2f43cba3-15f4-4170-8b22-36f666bffe9d nodeName:}" failed. No retries permitted until 2026-07-10 18:41:00.198855849 +0000 UTC m=+37.666774659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f4w65" (UID: "2f43cba3-15f4-4170-8b22-36f666bffe9d") : configmap "kube-rbac-proxy" not found
Jul 10 18:40:59.698988 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.698896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f43cba3-15f4-4170-8b22-36f666bffe9d-crio-socket\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:40:59.698988 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.698912    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxk7j\" (UniqueName: \"kubernetes.io/projected/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-api-access-rxk7j\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:40:59.699198 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.699016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:40:59.699198 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.699094    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f43cba3-15f4-4170-8b22-36f666bffe9d-crio-socket\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:40:59.699198 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:59.699141    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:40:59.699333 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:40:59.699231    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls podName:2f43cba3-15f4-4170-8b22-36f666bffe9d nodeName:}" failed. No retries permitted until 2026-07-10 18:41:00.199213389 +0000 UTC m=+37.667132205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f4w65" (UID: "2f43cba3-15f4-4170-8b22-36f666bffe9d") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:40:59.699460 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.699439    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f43cba3-15f4-4170-8b22-36f666bffe9d-data-volume\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:40:59.715802 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:40:59.715668    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxk7j\" (UniqueName: \"kubernetes.io/projected/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-api-access-rxk7j\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:00.088672 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.088637    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:41:00.088862 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.088677    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:41:00.088862 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.088767    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:41:00.089687 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.089660    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-vmp4z"]
Jul 10 18:41:00.091709 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.091691    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-vmp4z"
Jul 10 18:41:00.094082 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.094064    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:41:00.094705 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.094677    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:41:00.094705 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.094680    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:41:00.094896 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.094722    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s52z2\""
Jul 10 18:41:00.095114 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.095095    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hzzqm\""
Jul 10 18:41:00.095482 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.095457    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:41:00.099201 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.099183    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 10 18:41:00.099439 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.099415    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 10 18:41:00.099535 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.099466    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 10 18:41:00.100054 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.100011    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 10 18:41:00.100231 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.100215    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nvxvp\""
Jul 10 18:41:00.111027 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.111005    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-vmp4z"]
Jul 10 18:41:00.202672 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.202639    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:00.202843 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.202679    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97pk\" (UniqueName: \"kubernetes.io/projected/76522eee-f269-4f57-a732-b6f36cc67525-kube-api-access-b97pk\") pod \"service-ca-79f68b9f4f-vmp4z\" (UID: \"76522eee-f269-4f57-a732-b6f36cc67525\") " pod="openshift-service-ca/service-ca-79f68b9f4f-vmp4z"
Jul 10 18:41:00.202843 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:00.202786    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:41:00.202843 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.202826    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/76522eee-f269-4f57-a732-b6f36cc67525-signing-key\") pod \"service-ca-79f68b9f4f-vmp4z\" (UID: \"76522eee-f269-4f57-a732-b6f36cc67525\") " pod="openshift-service-ca/service-ca-79f68b9f4f-vmp4z"
Jul 10 18:41:00.202972 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:00.202872    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm podName:2f43cba3-15f4-4170-8b22-36f666bffe9d nodeName:}" failed. No retries permitted until 2026-07-10 18:41:01.20285066 +0000 UTC m=+38.670769473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f4w65" (UID: "2f43cba3-15f4-4170-8b22-36f666bffe9d") : configmap "kube-rbac-proxy" not found
Jul 10 18:41:00.203059 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.203036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:00.203179 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.203096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/76522eee-f269-4f57-a732-b6f36cc67525-signing-cabundle\") pod \"service-ca-79f68b9f4f-vmp4z\" (UID: \"76522eee-f269-4f57-a732-b6f36cc67525\") " pod="openshift-service-ca/service-ca-79f68b9f4f-vmp4z"
Jul 10 18:41:00.203277 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:00.203200    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:00.203277 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:00.203257    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls podName:2f43cba3-15f4-4170-8b22-36f666bffe9d nodeName:}" failed. No retries permitted until 2026-07-10 18:41:01.203240345 +0000 UTC m=+38.671159166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f4w65" (UID: "2f43cba3-15f4-4170-8b22-36f666bffe9d") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:00.281945 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.281909    2565 generic.go:356] "Generic (PLEG): container finished" podID="be7bd63d-2e6e-4570-abf3-272865bfef5e" containerID="94ea8524635996db4f44b696f81b88aa755eeaf63b68dd94e46961d939a172b6" exitCode=0
Jul 10 18:41:00.282118 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.281958    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvx67" event={"ID":"be7bd63d-2e6e-4570-abf3-272865bfef5e","Type":"ContainerDied","Data":"94ea8524635996db4f44b696f81b88aa755eeaf63b68dd94e46961d939a172b6"}
Jul 10 18:41:00.283407 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.283372    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8zfrf" event={"ID":"6d6e03d6-117c-4854-b76c-dba72a1a2f27","Type":"ContainerStarted","Data":"75411b59f32830bdc5ebdbf508d4e25be8960db7cc1150972a80efaca7fcdd51"}
Jul 10 18:41:00.304343 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.304318    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97pk\" (UniqueName: \"kubernetes.io/projected/76522eee-f269-4f57-a732-b6f36cc67525-kube-api-access-b97pk\") pod \"service-ca-79f68b9f4f-vmp4z\" (UID: \"76522eee-f269-4f57-a732-b6f36cc67525\") " pod="openshift-service-ca/service-ca-79f68b9f4f-vmp4z"
Jul 10 18:41:00.304458 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.304419    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/76522eee-f269-4f57-a732-b6f36cc67525-signing-key\") pod \"service-ca-79f68b9f4f-vmp4z\" (UID: \"76522eee-f269-4f57-a732-b6f36cc67525\") " pod="openshift-service-ca/service-ca-79f68b9f4f-vmp4z"
Jul 10 18:41:00.304527 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.304508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/76522eee-f269-4f57-a732-b6f36cc67525-signing-cabundle\") pod \"service-ca-79f68b9f4f-vmp4z\" (UID: \"76522eee-f269-4f57-a732-b6f36cc67525\") " pod="openshift-service-ca/service-ca-79f68b9f4f-vmp4z"
Jul 10 18:41:00.305264 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.305219    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/76522eee-f269-4f57-a732-b6f36cc67525-signing-cabundle\") pod \"service-ca-79f68b9f4f-vmp4z\" (UID: \"76522eee-f269-4f57-a732-b6f36cc67525\") " pod="openshift-service-ca/service-ca-79f68b9f4f-vmp4z"
Jul 10 18:41:00.307859 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.307834    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/76522eee-f269-4f57-a732-b6f36cc67525-signing-key\") pod \"service-ca-79f68b9f4f-vmp4z\" (UID: \"76522eee-f269-4f57-a732-b6f36cc67525\") " pod="openshift-service-ca/service-ca-79f68b9f4f-vmp4z"
Jul 10 18:41:00.317084 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.317034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97pk\" (UniqueName: \"kubernetes.io/projected/76522eee-f269-4f57-a732-b6f36cc67525-kube-api-access-b97pk\") pod \"service-ca-79f68b9f4f-vmp4z\" (UID: \"76522eee-f269-4f57-a732-b6f36cc67525\") " pod="openshift-service-ca/service-ca-79f68b9f4f-vmp4z"
Jul 10 18:41:00.424055 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.424025    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-vmp4z"
Jul 10 18:41:00.570829 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.570675    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-vmp4z"]
Jul 10 18:41:00.574799 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:41:00.574768    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76522eee_f269_4f57_a732_b6f36cc67525.slice/crio-e6f97efe6f2bd2e3a25d432fefe79b436eda98b5e03dc39f9428dc713ff5f139 WatchSource:0}: Error finding container e6f97efe6f2bd2e3a25d432fefe79b436eda98b5e03dc39f9428dc713ff5f139: Status 404 returned error can't find the container with id e6f97efe6f2bd2e3a25d432fefe79b436eda98b5e03dc39f9428dc713ff5f139
Jul 10 18:41:00.608530 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:00.608455    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:41:00.608662 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:00.608582    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:41:00.608662 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:00.608647    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls podName:52c473ba-b012-460e-89a1-e32d7b7dc013 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:02.608627788 +0000 UTC m=+40.076546611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls") pod "dns-default-gp9v9" (UID: "52c473ba-b012-460e-89a1-e32d7b7dc013") : secret "dns-default-metrics-tls" not found
Jul 10 18:41:01.214163 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:01.214112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:01.215086 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:01.214192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:01.215086 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:01.214271    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:01.215086 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:01.214311    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:41:01.215086 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:01.214333    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls podName:2f43cba3-15f4-4170-8b22-36f666bffe9d nodeName:}" failed. No retries permitted until 2026-07-10 18:41:03.214316057 +0000 UTC m=+40.682234892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f4w65" (UID: "2f43cba3-15f4-4170-8b22-36f666bffe9d") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:01.215086 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:01.214365    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm podName:2f43cba3-15f4-4170-8b22-36f666bffe9d nodeName:}" failed. No retries permitted until 2026-07-10 18:41:03.214348922 +0000 UTC m=+40.682267746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f4w65" (UID: "2f43cba3-15f4-4170-8b22-36f666bffe9d") : configmap "kube-rbac-proxy" not found
Jul 10 18:41:01.287417 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:01.287373    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-vmp4z" event={"ID":"76522eee-f269-4f57-a732-b6f36cc67525","Type":"ContainerStarted","Data":"e6f97efe6f2bd2e3a25d432fefe79b436eda98b5e03dc39f9428dc713ff5f139"}
Jul 10 18:41:01.291586 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:01.291265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvx67" event={"ID":"be7bd63d-2e6e-4570-abf3-272865bfef5e","Type":"ContainerStarted","Data":"5feaf58a6adde8ed908cc971d24cd5acad4ea462dc9049d849e9f160ac6a4578"}
Jul 10 18:41:02.295324 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:02.295082    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8zfrf" event={"ID":"6d6e03d6-117c-4854-b76c-dba72a1a2f27","Type":"ContainerStarted","Data":"11008f915f3b02b59f718fe9cf9ce69db5ce7032f42c17f54bc8531dbc8c9d83"}
Jul 10 18:41:02.313682 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:02.313623    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xvx67" podStartSLOduration=9.221286278000001 podStartE2EDuration="40.31360064s" totalImagesPullingTime="31.092314362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:01.338754086 +0000 UTC m=+38.806672963" watchObservedRunningTime="2026-07-10 18:41:02.31360064 +0000 UTC m=+39.781519473"
Jul 10 18:41:02.313902 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:02.313874    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8zfrf" podStartSLOduration=2.229381201 podStartE2EDuration="4.313867283s" totalImagesPullingTime="2.084486082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:02.312505031 +0000 UTC m=+39.780423863" watchObservedRunningTime="2026-07-10 18:41:02.313867283 +0000 UTC m=+39.781786117"
Jul 10 18:41:02.376532 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:02.376500    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vb9ww_48f95840-5a3c-4014-b1c5-f489153c94ce/dns-node-resolver/0.log"
Jul 10 18:41:02.626715 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:02.626627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:41:02.626892 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:02.626769    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:41:02.626892 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:02.626845    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls podName:52c473ba-b012-460e-89a1-e32d7b7dc013 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:06.626829554 +0000 UTC m=+44.094748373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls") pod "dns-default-gp9v9" (UID: "52c473ba-b012-460e-89a1-e32d7b7dc013") : secret "dns-default-metrics-tls" not found
Jul 10 18:41:03.176220 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:03.176192    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bzjdq_975a4c98-91a8-4277-b3ff-a5cd2da229cb/node-ca/0.log"
Jul 10 18:41:03.231517 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:03.231477    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:03.231653 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:03.231588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:03.231653 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:03.231609    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:41:03.231738 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:03.231671    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm podName:2f43cba3-15f4-4170-8b22-36f666bffe9d nodeName:}" failed. No retries permitted until 2026-07-10 18:41:07.231651696 +0000 UTC m=+44.699570530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f4w65" (UID: "2f43cba3-15f4-4170-8b22-36f666bffe9d") : configmap "kube-rbac-proxy" not found
Jul 10 18:41:03.231738 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:03.231690    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:03.231738 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:03.231734    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls podName:2f43cba3-15f4-4170-8b22-36f666bffe9d nodeName:}" failed. No retries permitted until 2026-07-10 18:41:07.231719798 +0000 UTC m=+44.699638627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f4w65" (UID: "2f43cba3-15f4-4170-8b22-36f666bffe9d") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:03.299402 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:03.299362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-vmp4z" event={"ID":"76522eee-f269-4f57-a732-b6f36cc67525","Type":"ContainerStarted","Data":"f4a86749fc4f8699145c9eea70d4d503024156d3864dbdc495e6f8109b0d3e21"}
Jul 10 18:41:03.323551 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:03.323496    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-vmp4z" podStartSLOduration=1.113588807 podStartE2EDuration="3.323478025s" totalImagesPullingTime="2.209889218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:03.322869083 +0000 UTC m=+40.790787936" watchObservedRunningTime="2026-07-10 18:41:03.323478025 +0000 UTC m=+40.791396859"
Jul 10 18:41:03.789601 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:03.789573    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8zfrf_6d6e03d6-117c-4854-b76c-dba72a1a2f27/serve-healthcheck-canary/0.log"
Jul 10 18:41:06.663982 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:06.663940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:41:06.664389 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:06.664055    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:41:06.664389 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:06.664104    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls podName:52c473ba-b012-460e-89a1-e32d7b7dc013 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:14.664091368 +0000 UTC m=+52.132010178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls") pod "dns-default-gp9v9" (UID: "52c473ba-b012-460e-89a1-e32d7b7dc013") : secret "dns-default-metrics-tls" not found
Jul 10 18:41:07.268850 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:07.268810    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:07.269023 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:07.268948    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:41:07.269023 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:07.269012    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm podName:2f43cba3-15f4-4170-8b22-36f666bffe9d nodeName:}" failed. No retries permitted until 2026-07-10 18:41:15.268994247 +0000 UTC m=+52.736913076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f4w65" (UID: "2f43cba3-15f4-4170-8b22-36f666bffe9d") : configmap "kube-rbac-proxy" not found
Jul 10 18:41:07.269023 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:07.268950    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:07.269170 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:07.269023    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:07.269170 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:07.269060    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls podName:2f43cba3-15f4-4170-8b22-36f666bffe9d nodeName:}" failed. No retries permitted until 2026-07-10 18:41:15.269049431 +0000 UTC m=+52.736968256 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f4w65" (UID: "2f43cba3-15f4-4170-8b22-36f666bffe9d") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:14.723657 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:14.723618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:41:14.726785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:14.726759    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52c473ba-b012-460e-89a1-e32d7b7dc013-metrics-tls\") pod \"dns-default-gp9v9\" (UID: \"52c473ba-b012-460e-89a1-e32d7b7dc013\") " pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:41:14.802972 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:14.802937    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:41:14.981806 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:14.981740    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gp9v9"]
Jul 10 18:41:14.985730 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:41:14.985702    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c473ba_b012_460e_89a1_e32d7b7dc013.slice/crio-a8ee3b35a19a14b3cd67a60dc16597b94351923059d97dcaa68aab4bf23eeb07 WatchSource:0}: Error finding container a8ee3b35a19a14b3cd67a60dc16597b94351923059d97dcaa68aab4bf23eeb07: Status 404 returned error can't find the container with id a8ee3b35a19a14b3cd67a60dc16597b94351923059d97dcaa68aab4bf23eeb07
Jul 10 18:41:15.328248 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:15.328208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:15.328441 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:15.328270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:15.328441 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:15.328373    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:15.328555 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:41:15.328444    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls podName:2f43cba3-15f4-4170-8b22-36f666bffe9d nodeName:}" failed. No retries permitted until 2026-07-10 18:41:31.328423631 +0000 UTC m=+68.796342466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f4w65" (UID: "2f43cba3-15f4-4170-8b22-36f666bffe9d") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:15.328906 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:15.328885    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f43cba3-15f4-4170-8b22-36f666bffe9d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:15.330865 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:15.330841    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gp9v9" event={"ID":"52c473ba-b012-460e-89a1-e32d7b7dc013","Type":"ContainerStarted","Data":"a8ee3b35a19a14b3cd67a60dc16597b94351923059d97dcaa68aab4bf23eeb07"}
Jul 10 18:41:18.342573 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:18.342530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gp9v9" event={"ID":"52c473ba-b012-460e-89a1-e32d7b7dc013","Type":"ContainerStarted","Data":"1e431e1258a4ec19d168255fc644424640c6ea663f64c9e9312b2f2514d429a1"}
Jul 10 18:41:18.342977 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:18.342627    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gp9v9" event={"ID":"52c473ba-b012-460e-89a1-e32d7b7dc013","Type":"ContainerStarted","Data":"f16088f5ac2fc1d6343ec99eb7032ecc3b487cd8049b6d99250a77880cadb2d7"}
Jul 10 18:41:18.342977 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:18.342778    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:41:19.931400 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:19.931332    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gp9v9" podStartSLOduration=19.621369963 podStartE2EDuration="21.931305014s" totalImagesPullingTime="2.309935051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:18.365891469 +0000 UTC m=+55.833810322" watchObservedRunningTime="2026-07-10 18:41:19.931305014 +0000 UTC m=+57.399223849"
Jul 10 18:41:19.932203 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:19.932180    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-fm74x"]
Jul 10 18:41:19.979107 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:19.979073    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"]
Jul 10 18:41:19.979280 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:19.979212    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-fm74x"
Jul 10 18:41:19.982502 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:19.982483    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7fd4x\""
Jul 10 18:41:19.982659 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:19.982647    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 10 18:41:19.982956 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:19.982939    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 10 18:41:20.009213 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.009182    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-554d49f96b-zqx6m"]
Jul 10 18:41:20.009303 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.009214    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.015305 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.015277    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 10 18:41:20.015305 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.015295    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 10 18:41:20.015464 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.015310    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 10 18:41:20.015662 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.015640    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 10 18:41:20.016756 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.016738    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 10 18:41:20.016846 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.016787    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 10 18:41:20.016846 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.016784    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 10 18:41:20.042445 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.042421    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-554d49f96b-zqx6m"]
Jul 10 18:41:20.042445 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.042446    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"]
Jul 10 18:41:20.042602 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.042458    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-fm74x"]
Jul 10 18:41:20.042602 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.042535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-554d49f96b-zqx6m"
Jul 10 18:41:20.047123 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.047105    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9cf6d\""
Jul 10 18:41:20.047236 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.047110    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 10 18:41:20.062510 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.062488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e81b456-090c-4ce7-832a-1d57a18344cb-nginx-conf\") pod \"networking-console-plugin-6cf856758c-fm74x\" (UID: \"2e81b456-090c-4ce7-832a-1d57a18344cb\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-fm74x"
Jul 10 18:41:20.062641 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.062520    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.062641 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.062542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxjn\" (UniqueName: \"kubernetes.io/projected/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-kube-api-access-dpxjn\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.062641 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.062593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a5fb618-4737-4eaf-9317-e9b5cc680a51-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-554d49f96b-zqx6m\" (UID: \"6a5fb618-4737-4eaf-9317-e9b5cc680a51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-554d49f96b-zqx6m"
Jul 10 18:41:20.062641 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.062631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2e81b456-090c-4ce7-832a-1d57a18344cb-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-fm74x\" (UID: \"2e81b456-090c-4ce7-832a-1d57a18344cb\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-fm74x"
Jul 10 18:41:20.062858 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.062686    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhdrd\" (UniqueName: \"kubernetes.io/projected/6a5fb618-4737-4eaf-9317-e9b5cc680a51-kube-api-access-nhdrd\") pod \"managed-serviceaccount-addon-agent-554d49f96b-zqx6m\" (UID: \"6a5fb618-4737-4eaf-9317-e9b5cc680a51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-554d49f96b-zqx6m"
Jul 10 18:41:20.062858 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.062721    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-ca\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.062858 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.062745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-hub\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.062858 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.062763    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.062858 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.062809    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.163608 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.163574    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-ca\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.163608 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.163614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-hub\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.163862 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.163638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.163862 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.163747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.163862 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.163842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e81b456-090c-4ce7-832a-1d57a18344cb-nginx-conf\") pod \"networking-console-plugin-6cf856758c-fm74x\" (UID: \"2e81b456-090c-4ce7-832a-1d57a18344cb\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-fm74x"
Jul 10 18:41:20.164019 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.163888    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.164019 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.163929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxjn\" (UniqueName: \"kubernetes.io/projected/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-kube-api-access-dpxjn\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.164019 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.163955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a5fb618-4737-4eaf-9317-e9b5cc680a51-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-554d49f96b-zqx6m\" (UID: \"6a5fb618-4737-4eaf-9317-e9b5cc680a51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-554d49f96b-zqx6m"
Jul 10 18:41:20.164252 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.164015    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2e81b456-090c-4ce7-832a-1d57a18344cb-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-fm74x\" (UID: \"2e81b456-090c-4ce7-832a-1d57a18344cb\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-fm74x"
Jul 10 18:41:20.164252 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.164049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhdrd\" (UniqueName: \"kubernetes.io/projected/6a5fb618-4737-4eaf-9317-e9b5cc680a51-kube-api-access-nhdrd\") pod \"managed-serviceaccount-addon-agent-554d49f96b-zqx6m\" (UID: \"6a5fb618-4737-4eaf-9317-e9b5cc680a51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-554d49f96b-zqx6m"
Jul 10 18:41:20.164927 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.164899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.165110 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.165081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e81b456-090c-4ce7-832a-1d57a18344cb-nginx-conf\") pod \"networking-console-plugin-6cf856758c-fm74x\" (UID: \"2e81b456-090c-4ce7-832a-1d57a18344cb\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-fm74x"
Jul 10 18:41:20.166418 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.166392    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-ca\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.166656 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.166630    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-hub\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.166924 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.166902    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.166971 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.166907    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a5fb618-4737-4eaf-9317-e9b5cc680a51-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-554d49f96b-zqx6m\" (UID: \"6a5fb618-4737-4eaf-9317-e9b5cc680a51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-554d49f96b-zqx6m"
Jul 10 18:41:20.167061 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.167039    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2e81b456-090c-4ce7-832a-1d57a18344cb-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-fm74x\" (UID: \"2e81b456-090c-4ce7-832a-1d57a18344cb\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-fm74x"
Jul 10 18:41:20.167281 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.167265    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.182045 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.181992    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhdrd\" (UniqueName: \"kubernetes.io/projected/6a5fb618-4737-4eaf-9317-e9b5cc680a51-kube-api-access-nhdrd\") pod \"managed-serviceaccount-addon-agent-554d49f96b-zqx6m\" (UID: \"6a5fb618-4737-4eaf-9317-e9b5cc680a51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-554d49f96b-zqx6m"
Jul 10 18:41:20.186526 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.186500    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxjn\" (UniqueName: \"kubernetes.io/projected/5d450741-94c2-48dc-9bfd-bc65f04cf7e9-kube-api-access-dpxjn\") pod \"cluster-proxy-proxy-agent-5b799dd789-gthld\" (UID: \"5d450741-94c2-48dc-9bfd-bc65f04cf7e9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.287548 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.287516    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-fm74x"
Jul 10 18:41:20.333313 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.333290    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"
Jul 10 18:41:20.350126 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.350100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-554d49f96b-zqx6m"
Jul 10 18:41:20.485633 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.485606    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-fm74x"]
Jul 10 18:41:20.489067 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:41:20.489045    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e81b456_090c_4ce7_832a_1d57a18344cb.slice/crio-9cb6ed0d26be29e0e3353f2e3fc60ae6f83b15bfa7614c35d7bb1b26ff9c9d29 WatchSource:0}: Error finding container 9cb6ed0d26be29e0e3353f2e3fc60ae6f83b15bfa7614c35d7bb1b26ff9c9d29: Status 404 returned error can't find the container with id 9cb6ed0d26be29e0e3353f2e3fc60ae6f83b15bfa7614c35d7bb1b26ff9c9d29
Jul 10 18:41:20.522603 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.522576    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld"]
Jul 10 18:41:20.525279 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:41:20.525255    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d450741_94c2_48dc_9bfd_bc65f04cf7e9.slice/crio-655d83feaeda5649f1ca677da130fd23f4cedb4115c8b810f957a86cbb3a1a12 WatchSource:0}: Error finding container 655d83feaeda5649f1ca677da130fd23f4cedb4115c8b810f957a86cbb3a1a12: Status 404 returned error can't find the container with id 655d83feaeda5649f1ca677da130fd23f4cedb4115c8b810f957a86cbb3a1a12
Jul 10 18:41:20.554485 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:20.554462    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-554d49f96b-zqx6m"]
Jul 10 18:41:20.557369 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:41:20.557347    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a5fb618_4737_4eaf_9317_e9b5cc680a51.slice/crio-a9b6588b59280d83d68d31d43f0cf0d64c3b55a7cec271b0317fe3e035ea14d1 WatchSource:0}: Error finding container a9b6588b59280d83d68d31d43f0cf0d64c3b55a7cec271b0317fe3e035ea14d1: Status 404 returned error can't find the container with id a9b6588b59280d83d68d31d43f0cf0d64c3b55a7cec271b0317fe3e035ea14d1
Jul 10 18:41:21.357780 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:21.357568    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-fm74x" event={"ID":"2e81b456-090c-4ce7-832a-1d57a18344cb","Type":"ContainerStarted","Data":"9cb6ed0d26be29e0e3353f2e3fc60ae6f83b15bfa7614c35d7bb1b26ff9c9d29"}
Jul 10 18:41:21.359514 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:21.359480    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-554d49f96b-zqx6m" event={"ID":"6a5fb618-4737-4eaf-9317-e9b5cc680a51","Type":"ContainerStarted","Data":"a9b6588b59280d83d68d31d43f0cf0d64c3b55a7cec271b0317fe3e035ea14d1"}
Jul 10 18:41:21.361124 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:21.361097    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld" event={"ID":"5d450741-94c2-48dc-9bfd-bc65f04cf7e9","Type":"ContainerStarted","Data":"655d83feaeda5649f1ca677da130fd23f4cedb4115c8b810f957a86cbb3a1a12"}
Jul 10 18:41:22.263561 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:22.263529    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-87nb6"
Jul 10 18:41:23.370728 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:23.370586    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-fm74x" event={"ID":"2e81b456-090c-4ce7-832a-1d57a18344cb","Type":"ContainerStarted","Data":"6499e2cd357c6881bbc01014bf61b73e74d0505701044c9e6b69463d43b1c1c3"}
Jul 10 18:41:23.372398 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:23.372368    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-554d49f96b-zqx6m" event={"ID":"6a5fb618-4737-4eaf-9317-e9b5cc680a51","Type":"ContainerStarted","Data":"1053f53fe8033d0e51e3a90d5d9f1c6b8df38247ce4284fa4b2377257c52e5d5"}
Jul 10 18:41:23.391736 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:23.391687    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-fm74x" podStartSLOduration=1.6381394679999999 podStartE2EDuration="4.391671196s" totalImagesPullingTime="2.753531728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:23.390932996 +0000 UTC m=+60.858851827" watchObservedRunningTime="2026-07-10 18:41:23.391671196 +0000 UTC m=+60.859590067"
Jul 10 18:41:23.407542 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:23.407493    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-554d49f96b-zqx6m" podStartSLOduration=1.68351492 podStartE2EDuration="4.407477622s" totalImagesPullingTime="2.723962702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:23.40642692 +0000 UTC m=+60.874345752" watchObservedRunningTime="2026-07-10 18:41:23.407477622 +0000 UTC m=+60.875396453"
Jul 10 18:41:25.384125 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:25.384081    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld" event={"ID":"5d450741-94c2-48dc-9bfd-bc65f04cf7e9","Type":"ContainerStarted","Data":"0b64fc208f960b39dea282bd7dc82eb36c715a82046b6ac7e01373ff2131317c"}
Jul 10 18:41:25.384125 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:25.384125    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld" event={"ID":"5d450741-94c2-48dc-9bfd-bc65f04cf7e9","Type":"ContainerStarted","Data":"92cf1bd8721323bbf4c529cbffaa3a65305236a5385bd767f2d9b1241bb6ae6b"}
Jul 10 18:41:25.384575 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:25.384158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld" event={"ID":"5d450741-94c2-48dc-9bfd-bc65f04cf7e9","Type":"ContainerStarted","Data":"0fe43e23f427067c2e92d850d3032074a82e844bc572509108a68a2d5023bf04"}
Jul 10 18:41:25.407449 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:25.407398    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b799dd789-gthld" podStartSLOduration=2.286143499 podStartE2EDuration="6.407383837s" totalImagesPullingTime="4.121240338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:25.405688799 +0000 UTC m=+62.873607631" watchObservedRunningTime="2026-07-10 18:41:25.407383837 +0000 UTC m=+62.875302669"
Jul 10 18:41:28.347597 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.347569    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gp9v9"
Jul 10 18:41:28.732459 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.732368    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqlz\" (UniqueName: \"kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz\") pod \"network-check-target-crm4n\" (UID: \"e265d96b-328a-4f5d-aceb-ecd5a32693c1\") " pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:41:28.736020 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.735996    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:41:28.745732 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.745704    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:41:28.755486 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.755462    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqlz\" (UniqueName: \"kubernetes.io/projected/e265d96b-328a-4f5d-aceb-ecd5a32693c1-kube-api-access-qqqlz\") pod \"network-check-target-crm4n\" (UID: \"e265d96b-328a-4f5d-aceb-ecd5a32693c1\") " pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:41:28.833606 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.833554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs\") pod \"network-metrics-daemon-dj7xt\" (UID: \"b90039f7-1ab3-4de0-81fe-4829c384ed89\") " pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:41:28.833791 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.833631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:41:28.836953 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.836932    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:41:28.838049 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.838032    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:41:28.845847 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.845824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b90039f7-1ab3-4de0-81fe-4829c384ed89-metrics-certs\") pod \"network-metrics-daemon-dj7xt\" (UID: \"b90039f7-1ab3-4de0-81fe-4829c384ed89\") " pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:41:28.845975 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.845955    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6730ada-8f92-421c-8768-422891b4c4f9-original-pull-secret\") pod \"global-pull-secret-syncer-sljz9\" (UID: \"d6730ada-8f92-421c-8768-422891b4c4f9\") " pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:41:28.901356 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.901326    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sljz9"
Jul 10 18:41:28.914518 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.914484    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s52z2\""
Jul 10 18:41:28.920320 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.920299    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hzzqm\""
Jul 10 18:41:28.921313 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.921292    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:41:28.927335 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:28.927315    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dj7xt"
Jul 10 18:41:29.050698 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:29.050663    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sljz9"]
Jul 10 18:41:29.054674 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:41:29.054649    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6730ada_8f92_421c_8768_422891b4c4f9.slice/crio-13c9dd87b7523df80e00d088512db302df487dd61c195a0d160498090c184607 WatchSource:0}: Error finding container 13c9dd87b7523df80e00d088512db302df487dd61c195a0d160498090c184607: Status 404 returned error can't find the container with id 13c9dd87b7523df80e00d088512db302df487dd61c195a0d160498090c184607
Jul 10 18:41:29.076355 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:29.076331    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dj7xt"]
Jul 10 18:41:29.078775 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:41:29.078751    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90039f7_1ab3_4de0_81fe_4829c384ed89.slice/crio-a02eff2ef72dd495273a4fa132d079f49de7851f8aff8a246e8c6a8f2b6a0b4d WatchSource:0}: Error finding container a02eff2ef72dd495273a4fa132d079f49de7851f8aff8a246e8c6a8f2b6a0b4d: Status 404 returned error can't find the container with id a02eff2ef72dd495273a4fa132d079f49de7851f8aff8a246e8c6a8f2b6a0b4d
Jul 10 18:41:29.096839 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:29.096818    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-crm4n"]
Jul 10 18:41:29.099763 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:41:29.099742    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode265d96b_328a_4f5d_aceb_ecd5a32693c1.slice/crio-165ce5ab69f51bb12332cb00f5370cce7dc42432b2b84ffd1e3c3b962060fa5d WatchSource:0}: Error finding container 165ce5ab69f51bb12332cb00f5370cce7dc42432b2b84ffd1e3c3b962060fa5d: Status 404 returned error can't find the container with id 165ce5ab69f51bb12332cb00f5370cce7dc42432b2b84ffd1e3c3b962060fa5d
Jul 10 18:41:29.399357 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:29.399321    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dj7xt" event={"ID":"b90039f7-1ab3-4de0-81fe-4829c384ed89","Type":"ContainerStarted","Data":"a02eff2ef72dd495273a4fa132d079f49de7851f8aff8a246e8c6a8f2b6a0b4d"}
Jul 10 18:41:29.400832 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:29.400808    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sljz9" event={"ID":"d6730ada-8f92-421c-8768-422891b4c4f9","Type":"ContainerStarted","Data":"13c9dd87b7523df80e00d088512db302df487dd61c195a0d160498090c184607"}
Jul 10 18:41:29.401853 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:29.401831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-crm4n" event={"ID":"e265d96b-328a-4f5d-aceb-ecd5a32693c1","Type":"ContainerStarted","Data":"165ce5ab69f51bb12332cb00f5370cce7dc42432b2b84ffd1e3c3b962060fa5d"}
Jul 10 18:41:30.408767 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:30.408722    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dj7xt" event={"ID":"b90039f7-1ab3-4de0-81fe-4829c384ed89","Type":"ContainerStarted","Data":"c9716dbda9daed90068ee67d323fe3dd61620136830bb36c848a95a7d3a22120"}
Jul 10 18:41:30.409257 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:30.408828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dj7xt" event={"ID":"b90039f7-1ab3-4de0-81fe-4829c384ed89","Type":"ContainerStarted","Data":"b6b5405c90e0d7b6dff4ff89b4fa0f5ce5ec6ec7ed1357299a05bff96841e955"}
Jul 10 18:41:30.429988 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:30.429925    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dj7xt" podStartSLOduration=66.552160876 podStartE2EDuration="1m7.429905546s" totalImagesPullingTime="877.74467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:30.429783661 +0000 UTC m=+67.897702522" watchObservedRunningTime="2026-07-10 18:41:30.429905546 +0000 UTC m=+67.897824380"
Jul 10 18:41:31.357696 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:31.357661    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:31.360541 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:31.360485    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f43cba3-15f4-4170-8b22-36f666bffe9d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f4w65\" (UID: \"2f43cba3-15f4-4170-8b22-36f666bffe9d\") " pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:31.554403 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:31.554365    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nfnkx\""
Jul 10 18:41:31.561387 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:31.561356    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f4w65"
Jul 10 18:41:34.366827 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:34.366804    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f4w65"]
Jul 10 18:41:34.370074 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:41:34.370048    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f43cba3_15f4_4170_8b22_36f666bffe9d.slice/crio-e92e94a39f8440fe7e801592e17da2ca369b8579a221f9a50fb97ec06c006582 WatchSource:0}: Error finding container e92e94a39f8440fe7e801592e17da2ca369b8579a221f9a50fb97ec06c006582: Status 404 returned error can't find the container with id e92e94a39f8440fe7e801592e17da2ca369b8579a221f9a50fb97ec06c006582
Jul 10 18:41:34.426810 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:34.426777    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f4w65" event={"ID":"2f43cba3-15f4-4170-8b22-36f666bffe9d","Type":"ContainerStarted","Data":"e92e94a39f8440fe7e801592e17da2ca369b8579a221f9a50fb97ec06c006582"}
Jul 10 18:41:34.428249 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:34.428217    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sljz9" event={"ID":"d6730ada-8f92-421c-8768-422891b4c4f9","Type":"ContainerStarted","Data":"bdc7f8aa70cf83064be99044e9d738b6cb43025477227a670c52f1de64601161"}
Jul 10 18:41:34.429701 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:34.429675    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-crm4n" event={"ID":"e265d96b-328a-4f5d-aceb-ecd5a32693c1","Type":"ContainerStarted","Data":"c5b3fbfece94d112d7db1a6beec98603fef8d2defb933a029d9900c96766e768"}
Jul 10 18:41:34.429839 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:34.429815    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:41:34.449414 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:34.446448    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sljz9" podStartSLOduration=66.267491585 podStartE2EDuration="1m11.446427971s" totalImagesPullingTime="5.178936386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:34.44499163 +0000 UTC m=+71.912910486" watchObservedRunningTime="2026-07-10 18:41:34.446427971 +0000 UTC m=+71.914346804"
Jul 10 18:41:34.464621 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:34.464572    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-crm4n" podStartSLOduration=67.341500408 podStartE2EDuration="1m12.464554972s" totalImagesPullingTime="5.123054564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:34.463248019 +0000 UTC m=+71.931166879" watchObservedRunningTime="2026-07-10 18:41:34.464554972 +0000 UTC m=+71.932473842"
Jul 10 18:41:38.446682 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:38.446645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f4w65" event={"ID":"2f43cba3-15f4-4170-8b22-36f666bffe9d","Type":"ContainerStarted","Data":"93a128794b0ed8bd31e9eb8799c02b495ab96584a41bc7476d5c0fe61c2bd522"}
Jul 10 18:41:39.452935 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:39.452891    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f4w65" event={"ID":"2f43cba3-15f4-4170-8b22-36f666bffe9d","Type":"ContainerStarted","Data":"6dd4965d74cc1ec8c72dafd5776025e240d1e2dc4d1fd689a82d7303be57c367"}
Jul 10 18:41:40.298052 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.298018    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"]
Jul 10 18:41:40.300891 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.300860    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.305311 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.305283    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-29gsc"]
Jul 10 18:41:40.305979 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.305675    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 10 18:41:40.305979 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.305828    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 10 18:41:40.305979 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.305827    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 10 18:41:40.305979 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.305889    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 10 18:41:40.306259 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.306116    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 10 18:41:40.306806 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.306781    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 10 18:41:40.307466 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.307450    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cbdkt\""
Jul 10 18:41:40.307855 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.307834    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.314929 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.314907    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 10 18:41:40.315078 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.315056    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5ggl7\""
Jul 10 18:41:40.315306 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.315290    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 10 18:41:40.316549 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.316374    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 10 18:41:40.332204 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.332141    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"]
Jul 10 18:41:40.429242 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429202    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.429242 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-tls\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.429481 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429268    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.429481 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/914043ab-d211-44c3-9376-f5d6f3d40374-root\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.429481 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429425    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-wtmp\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.429645 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429482    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.429645 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-accelerators-collector-config\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.429645 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/914043ab-d211-44c3-9376-f5d6f3d40374-sys\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.429645 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.429856 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429677    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.429856 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429711    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/914043ab-d211-44c3-9376-f5d6f3d40374-metrics-client-ca\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.429856 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429739    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkxnc\" (UniqueName: \"kubernetes.io/projected/914043ab-d211-44c3-9376-f5d6f3d40374-kube-api-access-xkxnc\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.429856 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.429856 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429792    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-textfile\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.429856 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.429821    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvqhc\" (UniqueName: \"kubernetes.io/projected/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-kube-api-access-xvqhc\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.531208 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531167    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-textfile\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.531670 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvqhc\" (UniqueName: \"kubernetes.io/projected/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-kube-api-access-xvqhc\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.531670 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531274    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.531670 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-tls\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.531670 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.531670 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/914043ab-d211-44c3-9376-f5d6f3d40374-root\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.531670 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-wtmp\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.531670 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531430    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.531670 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531456    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-accelerators-collector-config\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.531670 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/914043ab-d211-44c3-9376-f5d6f3d40374-sys\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.531670 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531521    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.531670 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.531670 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531563    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-textfile\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.531670 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/914043ab-d211-44c3-9376-f5d6f3d40374-metrics-client-ca\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.532340 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkxnc\" (UniqueName: \"kubernetes.io/projected/914043ab-d211-44c3-9376-f5d6f3d40374-kube-api-access-xkxnc\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.532340 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.531632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.532340 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.532081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/914043ab-d211-44c3-9376-f5d6f3d40374-sys\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.532340 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.532102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-accelerators-collector-config\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.532580 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.532549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/914043ab-d211-44c3-9376-f5d6f3d40374-metrics-client-ca\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.532712 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.532605    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/914043ab-d211-44c3-9376-f5d6f3d40374-root\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.532712 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.532705    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.532827 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.532742    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-wtmp\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.533189 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.533112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.533370 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.533346    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.534503 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.534467    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-tls\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.534503 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.534491    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/914043ab-d211-44c3-9376-f5d6f3d40374-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.534856 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.534840    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.535437 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.535417    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.541493 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.541472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvqhc\" (UniqueName: \"kubernetes.io/projected/8d0f4fd3-5c15-4f46-b466-5bed333a7dc7-kube-api-access-xvqhc\") pod \"kube-state-metrics-785778ff74-vfrhr\" (UID: \"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.541583 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.541503    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkxnc\" (UniqueName: \"kubernetes.io/projected/914043ab-d211-44c3-9376-f5d6f3d40374-kube-api-access-xkxnc\") pod \"node-exporter-29gsc\" (UID: \"914043ab-d211-44c3-9376-f5d6f3d40374\") " pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:40.617785 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.617614    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"
Jul 10 18:41:40.625719 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:40.625326    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-29gsc"
Jul 10 18:41:41.200923 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:41:41.200891    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914043ab_d211_44c3_9376_f5d6f3d40374.slice/crio-f6fa55bae00e7dcb157f4a81bbdcf52cc294653d1a01bf0c399f451f6a203035 WatchSource:0}: Error finding container f6fa55bae00e7dcb157f4a81bbdcf52cc294653d1a01bf0c399f451f6a203035: Status 404 returned error can't find the container with id f6fa55bae00e7dcb157f4a81bbdcf52cc294653d1a01bf0c399f451f6a203035
Jul 10 18:41:41.350124 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:41.350094    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-vfrhr"]
Jul 10 18:41:41.353534 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:41:41.353502    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0f4fd3_5c15_4f46_b466_5bed333a7dc7.slice/crio-6fb2b0f6ddac0e46c37fa585416e35f6db60e7f0e1813cdcae8aeb44d4c0ba1d WatchSource:0}: Error finding container 6fb2b0f6ddac0e46c37fa585416e35f6db60e7f0e1813cdcae8aeb44d4c0ba1d: Status 404 returned error can't find the container with id 6fb2b0f6ddac0e46c37fa585416e35f6db60e7f0e1813cdcae8aeb44d4c0ba1d
Jul 10 18:41:41.461609 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:41.461518    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29gsc" event={"ID":"914043ab-d211-44c3-9376-f5d6f3d40374","Type":"ContainerStarted","Data":"f6fa55bae00e7dcb157f4a81bbdcf52cc294653d1a01bf0c399f451f6a203035"}
Jul 10 18:41:41.462752 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:41.462722    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr" event={"ID":"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7","Type":"ContainerStarted","Data":"6fb2b0f6ddac0e46c37fa585416e35f6db60e7f0e1813cdcae8aeb44d4c0ba1d"}
Jul 10 18:41:41.464937 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:41.464914    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f4w65" event={"ID":"2f43cba3-15f4-4170-8b22-36f666bffe9d","Type":"ContainerStarted","Data":"99e2430f36043e240b6a3cafbaceb0c32fdf6e41b079b16bcfd28dd7a192365c"}
Jul 10 18:41:41.499315 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:41.499266    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f4w65" podStartSLOduration=35.757486868 podStartE2EDuration="42.49924199s" totalImagesPullingTime="6.741755122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:41.498355583 +0000 UTC m=+78.966274416" watchObservedRunningTime="2026-07-10 18:41:41.49924199 +0000 UTC m=+78.967160821"
Jul 10 18:41:42.470778 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:42.470692    2565 generic.go:356] "Generic (PLEG): container finished" podID="914043ab-d211-44c3-9376-f5d6f3d40374" containerID="f701fe6c4020d446bcfbe3ec4f52537fb41949c1a881e777c160cbcacce5d566" exitCode=0
Jul 10 18:41:42.471200 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:42.470784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29gsc" event={"ID":"914043ab-d211-44c3-9376-f5d6f3d40374","Type":"ContainerDied","Data":"f701fe6c4020d446bcfbe3ec4f52537fb41949c1a881e777c160cbcacce5d566"}
Jul 10 18:41:43.476662 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:43.476621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr" event={"ID":"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7","Type":"ContainerStarted","Data":"11b674d13f62dea1f23aa9b913b9a3d2c45de74d1c53c9820d22fce60d88fe5c"}
Jul 10 18:41:43.477120 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:43.476690    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr" event={"ID":"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7","Type":"ContainerStarted","Data":"1ac195e4d319291be1b817273c26c9cbfe08bcb45db0427a704cdc16ba2c1171"}
Jul 10 18:41:43.477120 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:43.476718    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr" event={"ID":"8d0f4fd3-5c15-4f46-b466-5bed333a7dc7","Type":"ContainerStarted","Data":"085ca13e86b83477ad9e534cc7b988c43ef509adff420f898ce474e8cc774994"}
Jul 10 18:41:43.478842 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:43.478814    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29gsc" event={"ID":"914043ab-d211-44c3-9376-f5d6f3d40374","Type":"ContainerStarted","Data":"6f48a730d0e2a28f519711336ea88d2c949d7fe3a4ee02f4ad29d2ed6a8d0ce3"}
Jul 10 18:41:43.478944 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:43.478865    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29gsc" event={"ID":"914043ab-d211-44c3-9376-f5d6f3d40374","Type":"ContainerStarted","Data":"b7fee07dc91076947273cfe43206c36f0151aa618dfff0ad192d266da8f7f658"}
Jul 10 18:41:43.501096 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:43.501051    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-vfrhr" podStartSLOduration=2.131974766 podStartE2EDuration="3.501035894s" totalImagesPullingTime="1.369061128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:43.499116642 +0000 UTC m=+80.967035474" watchObservedRunningTime="2026-07-10 18:41:43.501035894 +0000 UTC m=+80.968954726"
Jul 10 18:41:43.519026 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:43.518981    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-29gsc" podStartSLOduration=2.63522079 podStartE2EDuration="3.518965841s" totalImagesPullingTime="817.821325ms" totalInitContainerRuntime="65.923726ms" isStatefulPod=true podCreationTimestamp="2026-07-10 18:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:43.51800977 +0000 UTC m=+80.985928602" watchObservedRunningTime="2026-07-10 18:41:43.518965841 +0000 UTC m=+80.986884672"
Jul 10 18:41:44.867342 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:44.867216    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7"]
Jul 10 18:41:44.869249 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:44.869230    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7"
Jul 10 18:41:44.873099 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:44.873072    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 10 18:41:44.873394 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:44.873378    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7g9k5\""
Jul 10 18:41:44.882488 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:44.882465    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7"]
Jul 10 18:41:44.970320 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:44.970285    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl6s4\" (UniqueName: \"kubernetes.io/projected/33fdee38-87f5-4255-8f13-5ba6bae49a83-kube-api-access-dl6s4\") pod \"monitoring-plugin-5f68df7f58-6vpv7\" (UID: \"33fdee38-87f5-4255-8f13-5ba6bae49a83\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7"
Jul 10 18:41:44.970320 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:44.970330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33fdee38-87f5-4255-8f13-5ba6bae49a83-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-6vpv7\" (UID: \"33fdee38-87f5-4255-8f13-5ba6bae49a83\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7"
Jul 10 18:41:45.071620 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:45.071585    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33fdee38-87f5-4255-8f13-5ba6bae49a83-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-6vpv7\" (UID: \"33fdee38-87f5-4255-8f13-5ba6bae49a83\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7"
Jul 10 18:41:45.071775 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:45.071657    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl6s4\" (UniqueName: \"kubernetes.io/projected/33fdee38-87f5-4255-8f13-5ba6bae49a83-kube-api-access-dl6s4\") pod \"monitoring-plugin-5f68df7f58-6vpv7\" (UID: \"33fdee38-87f5-4255-8f13-5ba6bae49a83\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7"
Jul 10 18:41:45.073993 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:45.073972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/33fdee38-87f5-4255-8f13-5ba6bae49a83-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-6vpv7\" (UID: \"33fdee38-87f5-4255-8f13-5ba6bae49a83\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7"
Jul 10 18:41:45.080760 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:45.080732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl6s4\" (UniqueName: \"kubernetes.io/projected/33fdee38-87f5-4255-8f13-5ba6bae49a83-kube-api-access-dl6s4\") pod \"monitoring-plugin-5f68df7f58-6vpv7\" (UID: \"33fdee38-87f5-4255-8f13-5ba6bae49a83\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7"
Jul 10 18:41:45.177728 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:45.177646    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7"
Jul 10 18:41:45.351316 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:45.351282    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7"]
Jul 10 18:41:45.354381 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:41:45.354347    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33fdee38_87f5_4255_8f13_5ba6bae49a83.slice/crio-ce044249a09720c552c3f03badb66826bbb1b7e69c6b9e1731034365b39f6913 WatchSource:0}: Error finding container ce044249a09720c552c3f03badb66826bbb1b7e69c6b9e1731034365b39f6913: Status 404 returned error can't find the container with id ce044249a09720c552c3f03badb66826bbb1b7e69c6b9e1731034365b39f6913
Jul 10 18:41:45.487056 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:45.486962    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7" event={"ID":"33fdee38-87f5-4255-8f13-5ba6bae49a83","Type":"ContainerStarted","Data":"ce044249a09720c552c3f03badb66826bbb1b7e69c6b9e1731034365b39f6913"}
Jul 10 18:41:47.496484 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:47.496448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7" event={"ID":"33fdee38-87f5-4255-8f13-5ba6bae49a83","Type":"ContainerStarted","Data":"6f1060f9fe573d13b57996c8edfba051940fc31e05ea49f4b46b3d8d81b04466"}
Jul 10 18:41:47.496944 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:47.496627    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7"
Jul 10 18:41:47.501431 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:47.501408    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7"
Jul 10 18:41:47.518093 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:41:47.518048    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-6vpv7" podStartSLOduration=2.303606811 podStartE2EDuration="3.518006382s" totalImagesPullingTime="1.214399571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:47.516867917 +0000 UTC m=+84.984786750" watchObservedRunningTime="2026-07-10 18:41:47.518006382 +0000 UTC m=+84.985925213"
Jul 10 18:42:05.435713 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:42:05.435650    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-crm4n"
Jul 10 18:45:22.926966 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:45:22.926939    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 10 18:47:12.348427 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.348395    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t2tps"]
Jul 10 18:47:12.351428 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.351409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t2tps"
Jul 10 18:47:12.354453 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.354428    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:47:12.354586 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.354508    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-x24wb\""
Jul 10 18:47:12.354647 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.354634    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 10 18:47:12.369045 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.369012    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t2tps"]
Jul 10 18:47:12.489931 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.489897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19bc8be7-cfd2-4f0c-9f29-60c3248a3cef-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t2tps\" (UID: \"19bc8be7-cfd2-4f0c-9f29-60c3248a3cef\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t2tps"
Jul 10 18:47:12.490115 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.489943    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9wgx\" (UniqueName: \"kubernetes.io/projected/19bc8be7-cfd2-4f0c-9f29-60c3248a3cef-kube-api-access-s9wgx\") pod \"cert-manager-operator-controller-manager-55c444bf86-t2tps\" (UID: \"19bc8be7-cfd2-4f0c-9f29-60c3248a3cef\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t2tps"
Jul 10 18:47:12.590574 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.590530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19bc8be7-cfd2-4f0c-9f29-60c3248a3cef-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t2tps\" (UID: \"19bc8be7-cfd2-4f0c-9f29-60c3248a3cef\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t2tps"
Jul 10 18:47:12.590753 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.590588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9wgx\" (UniqueName: \"kubernetes.io/projected/19bc8be7-cfd2-4f0c-9f29-60c3248a3cef-kube-api-access-s9wgx\") pod \"cert-manager-operator-controller-manager-55c444bf86-t2tps\" (UID: \"19bc8be7-cfd2-4f0c-9f29-60c3248a3cef\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t2tps"
Jul 10 18:47:12.590921 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.590899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19bc8be7-cfd2-4f0c-9f29-60c3248a3cef-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t2tps\" (UID: \"19bc8be7-cfd2-4f0c-9f29-60c3248a3cef\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t2tps"
Jul 10 18:47:12.603449 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.603375    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9wgx\" (UniqueName: \"kubernetes.io/projected/19bc8be7-cfd2-4f0c-9f29-60c3248a3cef-kube-api-access-s9wgx\") pod \"cert-manager-operator-controller-manager-55c444bf86-t2tps\" (UID: \"19bc8be7-cfd2-4f0c-9f29-60c3248a3cef\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t2tps"
Jul 10 18:47:12.660334 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.660299    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t2tps"
Jul 10 18:47:12.798220 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.798189    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t2tps"]
Jul 10 18:47:12.800868 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:47:12.800840    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bc8be7_cfd2_4f0c_9f29_60c3248a3cef.slice/crio-8f574d2a68976d6bdf061e2a133c4620b630c7f588553ef4150f0b929e18e07d WatchSource:0}: Error finding container 8f574d2a68976d6bdf061e2a133c4620b630c7f588553ef4150f0b929e18e07d: Status 404 returned error can't find the container with id 8f574d2a68976d6bdf061e2a133c4620b630c7f588553ef4150f0b929e18e07d
Jul 10 18:47:12.803686 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:12.803669    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:47:13.602292 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:13.602255    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t2tps" event={"ID":"19bc8be7-cfd2-4f0c-9f29-60c3248a3cef","Type":"ContainerStarted","Data":"8f574d2a68976d6bdf061e2a133c4620b630c7f588553ef4150f0b929e18e07d"}
Jul 10 18:47:15.612369 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:15.612333    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t2tps" event={"ID":"19bc8be7-cfd2-4f0c-9f29-60c3248a3cef","Type":"ContainerStarted","Data":"b4687af82ce7b852e2f76c3210182425c089eff688e1086ccf7fd2c7ec97952d"}
Jul 10 18:47:15.635492 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:15.635385    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t2tps" podStartSLOduration=1.062597535 podStartE2EDuration="3.635367213s" totalImagesPullingTime="2.572769678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:47:15.633546157 +0000 UTC m=+413.101464989" watchObservedRunningTime="2026-07-10 18:47:15.635367213 +0000 UTC m=+413.103286044"
Jul 10 18:47:19.589226 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:19.589184    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lv42l"]
Jul 10 18:47:19.593391 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:19.593360    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lv42l"
Jul 10 18:47:19.596890 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:19.596860    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 10 18:47:19.598110 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:19.598091    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 10 18:47:19.598277 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:19.598096    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-rbfwm\""
Jul 10 18:47:19.601699 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:19.601676    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lv42l"]
Jul 10 18:47:19.744802 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:19.744754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g292\" (UniqueName: \"kubernetes.io/projected/11fc16d9-02b0-43a7-b73b-ab22821c1e24-kube-api-access-2g292\") pod \"cert-manager-cainjector-559cdd8c87-lv42l\" (UID: \"11fc16d9-02b0-43a7-b73b-ab22821c1e24\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lv42l"
Jul 10 18:47:19.744802 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:19.744806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11fc16d9-02b0-43a7-b73b-ab22821c1e24-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lv42l\" (UID: \"11fc16d9-02b0-43a7-b73b-ab22821c1e24\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lv42l"
Jul 10 18:47:19.845300 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:19.845209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g292\" (UniqueName: \"kubernetes.io/projected/11fc16d9-02b0-43a7-b73b-ab22821c1e24-kube-api-access-2g292\") pod \"cert-manager-cainjector-559cdd8c87-lv42l\" (UID: \"11fc16d9-02b0-43a7-b73b-ab22821c1e24\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lv42l"
Jul 10 18:47:19.845300 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:19.845260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11fc16d9-02b0-43a7-b73b-ab22821c1e24-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lv42l\" (UID: \"11fc16d9-02b0-43a7-b73b-ab22821c1e24\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lv42l"
Jul 10 18:47:19.853912 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:19.853884    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11fc16d9-02b0-43a7-b73b-ab22821c1e24-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lv42l\" (UID: \"11fc16d9-02b0-43a7-b73b-ab22821c1e24\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lv42l"
Jul 10 18:47:19.854084 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:19.854068    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g292\" (UniqueName: \"kubernetes.io/projected/11fc16d9-02b0-43a7-b73b-ab22821c1e24-kube-api-access-2g292\") pod \"cert-manager-cainjector-559cdd8c87-lv42l\" (UID: \"11fc16d9-02b0-43a7-b73b-ab22821c1e24\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lv42l"
Jul 10 18:47:19.903133 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:19.903087    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lv42l"
Jul 10 18:47:20.030930 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:20.030893    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lv42l"]
Jul 10 18:47:20.034083 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:47:20.034055    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11fc16d9_02b0_43a7_b73b_ab22821c1e24.slice/crio-071af5d27aea4ca759b0cf0646be0dca18f0574f3b678219142cceb24a7a7d5f WatchSource:0}: Error finding container 071af5d27aea4ca759b0cf0646be0dca18f0574f3b678219142cceb24a7a7d5f: Status 404 returned error can't find the container with id 071af5d27aea4ca759b0cf0646be0dca18f0574f3b678219142cceb24a7a7d5f
Jul 10 18:47:20.630988 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:20.630950    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lv42l" event={"ID":"11fc16d9-02b0-43a7-b73b-ab22821c1e24","Type":"ContainerStarted","Data":"071af5d27aea4ca759b0cf0646be0dca18f0574f3b678219142cceb24a7a7d5f"}
Jul 10 18:47:23.645465 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:23.645429    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lv42l" event={"ID":"11fc16d9-02b0-43a7-b73b-ab22821c1e24","Type":"ContainerStarted","Data":"4bc910abbfd93938a7b84e0c2b254f455fe0cfb6d994f399c97c5ff2903130d1"}
Jul 10 18:47:23.666884 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:23.666830    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lv42l" podStartSLOduration=1.981491197 podStartE2EDuration="4.666814538s" totalImagesPullingTime="2.685323341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:47:23.662228956 +0000 UTC m=+421.130147787" watchObservedRunningTime="2026-07-10 18:47:23.666814538 +0000 UTC m=+421.134733371"
Jul 10 18:47:24.185996 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:24.185958    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sl7rf"]
Jul 10 18:47:24.189309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:24.189291    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sl7rf"
Jul 10 18:47:24.194202 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:24.194183    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6mw8j\""
Jul 10 18:47:24.198379 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:24.198356    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sl7rf"]
Jul 10 18:47:24.280276 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:24.280231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f116cce5-8ff9-42b9-ac2f-f9afb49d28ff-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sl7rf\" (UID: \"f116cce5-8ff9-42b9-ac2f-f9afb49d28ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sl7rf"
Jul 10 18:47:24.280456 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:24.280315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r29hj\" (UniqueName: \"kubernetes.io/projected/f116cce5-8ff9-42b9-ac2f-f9afb49d28ff-kube-api-access-r29hj\") pod \"cert-manager-webhook-6486ffd997-sl7rf\" (UID: \"f116cce5-8ff9-42b9-ac2f-f9afb49d28ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sl7rf"
Jul 10 18:47:24.381218 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:24.381173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f116cce5-8ff9-42b9-ac2f-f9afb49d28ff-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sl7rf\" (UID: \"f116cce5-8ff9-42b9-ac2f-f9afb49d28ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sl7rf"
Jul 10 18:47:24.381218 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:24.381225    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r29hj\" (UniqueName: \"kubernetes.io/projected/f116cce5-8ff9-42b9-ac2f-f9afb49d28ff-kube-api-access-r29hj\") pod \"cert-manager-webhook-6486ffd997-sl7rf\" (UID: \"f116cce5-8ff9-42b9-ac2f-f9afb49d28ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sl7rf"
Jul 10 18:47:24.389684 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:24.389644    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f116cce5-8ff9-42b9-ac2f-f9afb49d28ff-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sl7rf\" (UID: \"f116cce5-8ff9-42b9-ac2f-f9afb49d28ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sl7rf"
Jul 10 18:47:24.389823 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:24.389708    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r29hj\" (UniqueName: \"kubernetes.io/projected/f116cce5-8ff9-42b9-ac2f-f9afb49d28ff-kube-api-access-r29hj\") pod \"cert-manager-webhook-6486ffd997-sl7rf\" (UID: \"f116cce5-8ff9-42b9-ac2f-f9afb49d28ff\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sl7rf"
Jul 10 18:47:24.497941 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:24.497830    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sl7rf"
Jul 10 18:47:24.632418 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:24.632382    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sl7rf"]
Jul 10 18:47:24.635374 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:47:24.635336    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf116cce5_8ff9_42b9_ac2f_f9afb49d28ff.slice/crio-18ca5055cb9d8bccd81563902f62cc03ffd258862022719cad876da36ff089d9 WatchSource:0}: Error finding container 18ca5055cb9d8bccd81563902f62cc03ffd258862022719cad876da36ff089d9: Status 404 returned error can't find the container with id 18ca5055cb9d8bccd81563902f62cc03ffd258862022719cad876da36ff089d9
Jul 10 18:47:24.650597 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:24.650566    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sl7rf" event={"ID":"f116cce5-8ff9-42b9-ac2f-f9afb49d28ff","Type":"ContainerStarted","Data":"18ca5055cb9d8bccd81563902f62cc03ffd258862022719cad876da36ff089d9"}
Jul 10 18:47:25.656325 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:25.656289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sl7rf" event={"ID":"f116cce5-8ff9-42b9-ac2f-f9afb49d28ff","Type":"ContainerStarted","Data":"bae097ba50cf7487955ce78f1fd16d499dfb3830c285e78c825fee1d9b0d0d85"}
Jul 10 18:47:25.656762 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:25.656338    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sl7rf"
Jul 10 18:47:25.675009 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:25.674951    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-sl7rf" podStartSLOduration=1.674932292 podStartE2EDuration="1.674932292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:47:25.674436969 +0000 UTC m=+423.142355800" watchObservedRunningTime="2026-07-10 18:47:25.674932292 +0000 UTC m=+423.142851123"
Jul 10 18:47:31.664225 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:47:31.664194    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sl7rf"
Jul 10 18:48:08.863170 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:08.863116    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"]
Jul 10 18:48:08.865196 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:08.865180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:08.871044 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:08.871014    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 10 18:48:08.871223 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:08.871039    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-r2nwg\""
Jul 10 18:48:08.871223 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:08.871065    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:48:08.871223 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:08.871100    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 10 18:48:08.871223 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:08.871140    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 10 18:48:08.874246 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:08.874214    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 10 18:48:08.888711 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:08.888679    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"]
Jul 10 18:48:08.919427 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:08.919400    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/416e9ca7-99be-40d1-aba0-dd3ff1ecee2a-cert\") pod \"lws-controller-manager-849cc6f9d7-gwqhq\" (UID: \"416e9ca7-99be-40d1-aba0-dd3ff1ecee2a\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:08.919574 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:08.919444    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn5kh\" (UniqueName: \"kubernetes.io/projected/416e9ca7-99be-40d1-aba0-dd3ff1ecee2a-kube-api-access-sn5kh\") pod \"lws-controller-manager-849cc6f9d7-gwqhq\" (UID: \"416e9ca7-99be-40d1-aba0-dd3ff1ecee2a\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:08.919574 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:08.919477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/416e9ca7-99be-40d1-aba0-dd3ff1ecee2a-manager-config\") pod \"lws-controller-manager-849cc6f9d7-gwqhq\" (UID: \"416e9ca7-99be-40d1-aba0-dd3ff1ecee2a\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:08.919574 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:08.919514    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/416e9ca7-99be-40d1-aba0-dd3ff1ecee2a-metrics-cert\") pod \"lws-controller-manager-849cc6f9d7-gwqhq\" (UID: \"416e9ca7-99be-40d1-aba0-dd3ff1ecee2a\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:09.020792 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:09.020758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn5kh\" (UniqueName: \"kubernetes.io/projected/416e9ca7-99be-40d1-aba0-dd3ff1ecee2a-kube-api-access-sn5kh\") pod \"lws-controller-manager-849cc6f9d7-gwqhq\" (UID: \"416e9ca7-99be-40d1-aba0-dd3ff1ecee2a\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:09.021003 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:09.020812    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/416e9ca7-99be-40d1-aba0-dd3ff1ecee2a-manager-config\") pod \"lws-controller-manager-849cc6f9d7-gwqhq\" (UID: \"416e9ca7-99be-40d1-aba0-dd3ff1ecee2a\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:09.021003 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:09.020829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/416e9ca7-99be-40d1-aba0-dd3ff1ecee2a-metrics-cert\") pod \"lws-controller-manager-849cc6f9d7-gwqhq\" (UID: \"416e9ca7-99be-40d1-aba0-dd3ff1ecee2a\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:09.021003 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:09.020861    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/416e9ca7-99be-40d1-aba0-dd3ff1ecee2a-cert\") pod \"lws-controller-manager-849cc6f9d7-gwqhq\" (UID: \"416e9ca7-99be-40d1-aba0-dd3ff1ecee2a\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:09.021616 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:09.021588    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/416e9ca7-99be-40d1-aba0-dd3ff1ecee2a-manager-config\") pod \"lws-controller-manager-849cc6f9d7-gwqhq\" (UID: \"416e9ca7-99be-40d1-aba0-dd3ff1ecee2a\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:09.023367 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:09.023341    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/416e9ca7-99be-40d1-aba0-dd3ff1ecee2a-cert\") pod \"lws-controller-manager-849cc6f9d7-gwqhq\" (UID: \"416e9ca7-99be-40d1-aba0-dd3ff1ecee2a\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:09.023511 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:09.023491    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/416e9ca7-99be-40d1-aba0-dd3ff1ecee2a-metrics-cert\") pod \"lws-controller-manager-849cc6f9d7-gwqhq\" (UID: \"416e9ca7-99be-40d1-aba0-dd3ff1ecee2a\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:09.031698 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:09.031672    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn5kh\" (UniqueName: \"kubernetes.io/projected/416e9ca7-99be-40d1-aba0-dd3ff1ecee2a-kube-api-access-sn5kh\") pod \"lws-controller-manager-849cc6f9d7-gwqhq\" (UID: \"416e9ca7-99be-40d1-aba0-dd3ff1ecee2a\") " pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:09.175275 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:09.175137    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:09.320215 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:09.320186    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"]
Jul 10 18:48:09.321781 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:48:09.321758    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416e9ca7_99be_40d1_aba0_dd3ff1ecee2a.slice/crio-2cbf25499a8534795eab26a2fd82b72368fe0a7eb2d0317abd7d535544b8dc83 WatchSource:0}: Error finding container 2cbf25499a8534795eab26a2fd82b72368fe0a7eb2d0317abd7d535544b8dc83: Status 404 returned error can't find the container with id 2cbf25499a8534795eab26a2fd82b72368fe0a7eb2d0317abd7d535544b8dc83
Jul 10 18:48:09.824881 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:09.824849    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq" event={"ID":"416e9ca7-99be-40d1-aba0-dd3ff1ecee2a","Type":"ContainerStarted","Data":"2cbf25499a8534795eab26a2fd82b72368fe0a7eb2d0317abd7d535544b8dc83"}
Jul 10 18:48:12.839983 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:12.839945    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq" event={"ID":"416e9ca7-99be-40d1-aba0-dd3ff1ecee2a","Type":"ContainerStarted","Data":"ecd098ac78eca244f472a180383942937db485c53ef938382723586acb171419"}
Jul 10 18:48:12.840406 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:12.840071    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:12.857594 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:12.857545    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq" podStartSLOduration=2.273974547 podStartE2EDuration="4.857528493s" totalImagesPullingTime="2.583553946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:48:12.85727526 +0000 UTC m=+470.325194129" watchObservedRunningTime="2026-07-10 18:48:12.857528493 +0000 UTC m=+470.325447324"
Jul 10 18:48:23.845522 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:23.845488    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-849cc6f9d7-gwqhq"
Jul 10 18:48:41.474700 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:41.474666    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-zcjhd"]
Jul 10 18:48:41.480996 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:41.480974    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-zcjhd"
Jul 10 18:48:41.483948 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:41.483923    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 10 18:48:41.484067 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:41.483923    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-ndnkz\""
Jul 10 18:48:41.484322 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:41.484307    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 10 18:48:41.485156 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:41.485130    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 10 18:48:41.492594 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:41.492569    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-zcjhd"]
Jul 10 18:48:41.580551 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:41.580507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mh9v\" (UniqueName: \"kubernetes.io/projected/71ee2e20-47f1-4ef6-aafd-ee4d850840d4-kube-api-access-6mh9v\") pod \"dns-operator-controller-manager-65b49595d7-zcjhd\" (UID: \"71ee2e20-47f1-4ef6-aafd-ee4d850840d4\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-zcjhd"
Jul 10 18:48:41.681891 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:41.681854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mh9v\" (UniqueName: \"kubernetes.io/projected/71ee2e20-47f1-4ef6-aafd-ee4d850840d4-kube-api-access-6mh9v\") pod \"dns-operator-controller-manager-65b49595d7-zcjhd\" (UID: \"71ee2e20-47f1-4ef6-aafd-ee4d850840d4\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-zcjhd"
Jul 10 18:48:41.695653 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:41.695622    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mh9v\" (UniqueName: \"kubernetes.io/projected/71ee2e20-47f1-4ef6-aafd-ee4d850840d4-kube-api-access-6mh9v\") pod \"dns-operator-controller-manager-65b49595d7-zcjhd\" (UID: \"71ee2e20-47f1-4ef6-aafd-ee4d850840d4\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-zcjhd"
Jul 10 18:48:41.790638 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:41.790604    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-zcjhd"
Jul 10 18:48:41.922735 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:41.922616    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-zcjhd"]
Jul 10 18:48:41.925051 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:48:41.925030    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ee2e20_47f1_4ef6_aafd_ee4d850840d4.slice/crio-d529d332d35d4d861acab368eca05f5501a0e155aa1516fb1726eb9a4ea4f888 WatchSource:0}: Error finding container d529d332d35d4d861acab368eca05f5501a0e155aa1516fb1726eb9a4ea4f888: Status 404 returned error can't find the container with id d529d332d35d4d861acab368eca05f5501a0e155aa1516fb1726eb9a4ea4f888
Jul 10 18:48:41.960782 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:41.960749    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-zcjhd" event={"ID":"71ee2e20-47f1-4ef6-aafd-ee4d850840d4","Type":"ContainerStarted","Data":"d529d332d35d4d861acab368eca05f5501a0e155aa1516fb1726eb9a4ea4f888"}
Jul 10 18:48:43.975953 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:43.975915    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-zcjhd" event={"ID":"71ee2e20-47f1-4ef6-aafd-ee4d850840d4","Type":"ContainerStarted","Data":"b953976eef40d2991d6bf08278f09a8a4372008265e620cdb351903f9098d254"}
Jul 10 18:48:43.976366 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:43.976023    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-zcjhd"
Jul 10 18:48:44.001760 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:44.001710    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-zcjhd" podStartSLOduration=1.816963925 podStartE2EDuration="3.001694091s" totalImagesPullingTime="1.184730166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:48:44.000960456 +0000 UTC m=+501.468879314" watchObservedRunningTime="2026-07-10 18:48:44.001694091 +0000 UTC m=+501.469612921"
Jul 10 18:48:45.529051 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:45.529013    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj"]
Jul 10 18:48:45.532187 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:45.532170    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj"
Jul 10 18:48:45.535094 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:45.535077    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-54ll6\""
Jul 10 18:48:45.545642 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:45.545621    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj"]
Jul 10 18:48:45.617341 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:45.617304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lts5b\" (UniqueName: \"kubernetes.io/projected/b421cb67-e157-41d4-b071-5247ed017ff6-kube-api-access-lts5b\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj\" (UID: \"b421cb67-e157-41d4-b071-5247ed017ff6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj"
Jul 10 18:48:45.617521 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:45.617373    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b421cb67-e157-41d4-b071-5247ed017ff6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj\" (UID: \"b421cb67-e157-41d4-b071-5247ed017ff6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj"
Jul 10 18:48:45.718176 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:45.718111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b421cb67-e157-41d4-b071-5247ed017ff6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj\" (UID: \"b421cb67-e157-41d4-b071-5247ed017ff6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj"
Jul 10 18:48:45.718347 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:45.718210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lts5b\" (UniqueName: \"kubernetes.io/projected/b421cb67-e157-41d4-b071-5247ed017ff6-kube-api-access-lts5b\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj\" (UID: \"b421cb67-e157-41d4-b071-5247ed017ff6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj"
Jul 10 18:48:45.718533 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:45.718515    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b421cb67-e157-41d4-b071-5247ed017ff6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj\" (UID: \"b421cb67-e157-41d4-b071-5247ed017ff6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj"
Jul 10 18:48:45.728027 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:45.727997    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lts5b\" (UniqueName: \"kubernetes.io/projected/b421cb67-e157-41d4-b071-5247ed017ff6-kube-api-access-lts5b\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj\" (UID: \"b421cb67-e157-41d4-b071-5247ed017ff6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj"
Jul 10 18:48:45.841901 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:45.841807    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj"
Jul 10 18:48:46.010101 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:46.010064    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj"]
Jul 10 18:48:46.014637 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:48:46.014599    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb421cb67_e157_41d4_b071_5247ed017ff6.slice/crio-7d9909e88786b058c0a8dbd79283a252b51b05e2093ffe1d86bf7ada8b2179fc WatchSource:0}: Error finding container 7d9909e88786b058c0a8dbd79283a252b51b05e2093ffe1d86bf7ada8b2179fc: Status 404 returned error can't find the container with id 7d9909e88786b058c0a8dbd79283a252b51b05e2093ffe1d86bf7ada8b2179fc
Jul 10 18:48:46.997337 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:46.997301    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj" event={"ID":"b421cb67-e157-41d4-b071-5247ed017ff6","Type":"ContainerStarted","Data":"7d9909e88786b058c0a8dbd79283a252b51b05e2093ffe1d86bf7ada8b2179fc"}
Jul 10 18:48:50.013169 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:50.013117    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj" event={"ID":"b421cb67-e157-41d4-b071-5247ed017ff6","Type":"ContainerStarted","Data":"c5d31d65c1b588c8f47e05a72bfdab4678d41acde478bb0a05655bb552e71342"}
Jul 10 18:48:50.013544 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:50.013295    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj"
Jul 10 18:48:50.035839 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:50.035790    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj" podStartSLOduration=1.253623157 podStartE2EDuration="5.035773652s" totalImagesPullingTime="3.782150495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:48:50.033731321 +0000 UTC m=+507.501650152" watchObservedRunningTime="2026-07-10 18:48:50.035773652 +0000 UTC m=+507.503692484"
Jul 10 18:48:54.982296 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:48:54.982265    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-zcjhd"
Jul 10 18:49:01.019697 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:01.019660    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj"
Jul 10 18:49:36.650879 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:36.650801    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c89bb85-xv7jv"]
Jul 10 18:49:36.652955 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:36.652928    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-xv7jv"
Jul 10 18:49:36.655733 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:36.655717    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mv7cp\""
Jul 10 18:49:36.661759 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:36.661724    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-xv7jv"]
Jul 10 18:49:36.712516 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:36.712477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r676p\" (UniqueName: \"kubernetes.io/projected/bdaecc62-345e-40da-96ef-c76e13c6d2e9-kube-api-access-r676p\") pod \"authorino-b6c89bb85-xv7jv\" (UID: \"bdaecc62-345e-40da-96ef-c76e13c6d2e9\") " pod="kuadrant-system/authorino-b6c89bb85-xv7jv"
Jul 10 18:49:36.758029 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:36.757994    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-p4wg4"]
Jul 10 18:49:36.760241 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:36.760223    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-p4wg4"
Jul 10 18:49:36.770037 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:36.770012    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-p4wg4"]
Jul 10 18:49:36.813831 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:36.813793    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q876f\" (UniqueName: \"kubernetes.io/projected/c797d13b-d937-4080-95c9-c5b4df07eb35-kube-api-access-q876f\") pod \"authorino-8df697c9f-p4wg4\" (UID: \"c797d13b-d937-4080-95c9-c5b4df07eb35\") " pod="kuadrant-system/authorino-8df697c9f-p4wg4"
Jul 10 18:49:36.814005 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:36.813876    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r676p\" (UniqueName: \"kubernetes.io/projected/bdaecc62-345e-40da-96ef-c76e13c6d2e9-kube-api-access-r676p\") pod \"authorino-b6c89bb85-xv7jv\" (UID: \"bdaecc62-345e-40da-96ef-c76e13c6d2e9\") " pod="kuadrant-system/authorino-b6c89bb85-xv7jv"
Jul 10 18:49:36.821664 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:36.821643    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r676p\" (UniqueName: \"kubernetes.io/projected/bdaecc62-345e-40da-96ef-c76e13c6d2e9-kube-api-access-r676p\") pod \"authorino-b6c89bb85-xv7jv\" (UID: \"bdaecc62-345e-40da-96ef-c76e13c6d2e9\") " pod="kuadrant-system/authorino-b6c89bb85-xv7jv"
Jul 10 18:49:36.914659 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:36.914568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q876f\" (UniqueName: \"kubernetes.io/projected/c797d13b-d937-4080-95c9-c5b4df07eb35-kube-api-access-q876f\") pod \"authorino-8df697c9f-p4wg4\" (UID: \"c797d13b-d937-4080-95c9-c5b4df07eb35\") " pod="kuadrant-system/authorino-8df697c9f-p4wg4"
Jul 10 18:49:36.922956 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:36.922927    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q876f\" (UniqueName: \"kubernetes.io/projected/c797d13b-d937-4080-95c9-c5b4df07eb35-kube-api-access-q876f\") pod \"authorino-8df697c9f-p4wg4\" (UID: \"c797d13b-d937-4080-95c9-c5b4df07eb35\") " pod="kuadrant-system/authorino-8df697c9f-p4wg4"
Jul 10 18:49:36.962946 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:36.962910    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-xv7jv"
Jul 10 18:49:37.069423 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:37.069392    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-p4wg4"
Jul 10 18:49:37.096042 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:37.095954    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-xv7jv"]
Jul 10 18:49:37.098454 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:49:37.098429    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdaecc62_345e_40da_96ef_c76e13c6d2e9.slice/crio-0c17956ff1865fb8818021d98b7e058a445979eb3e9f7874d405a33efeef76dd WatchSource:0}: Error finding container 0c17956ff1865fb8818021d98b7e058a445979eb3e9f7874d405a33efeef76dd: Status 404 returned error can't find the container with id 0c17956ff1865fb8818021d98b7e058a445979eb3e9f7874d405a33efeef76dd
Jul 10 18:49:37.206385 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:37.206292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-xv7jv" event={"ID":"bdaecc62-345e-40da-96ef-c76e13c6d2e9","Type":"ContainerStarted","Data":"0c17956ff1865fb8818021d98b7e058a445979eb3e9f7874d405a33efeef76dd"}
Jul 10 18:49:37.210709 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:37.210684    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-p4wg4"]
Jul 10 18:49:37.213688 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:49:37.213663    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc797d13b_d937_4080_95c9_c5b4df07eb35.slice/crio-cf543493dae23cb0c7b9de3591631f03b2a363bfc35bcf0184b4648614033176 WatchSource:0}: Error finding container cf543493dae23cb0c7b9de3591631f03b2a363bfc35bcf0184b4648614033176: Status 404 returned error can't find the container with id cf543493dae23cb0c7b9de3591631f03b2a363bfc35bcf0184b4648614033176
Jul 10 18:49:38.216944 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:38.216881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-p4wg4" event={"ID":"c797d13b-d937-4080-95c9-c5b4df07eb35","Type":"ContainerStarted","Data":"cf543493dae23cb0c7b9de3591631f03b2a363bfc35bcf0184b4648614033176"}
Jul 10 18:49:40.226528 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:40.226487    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-p4wg4" event={"ID":"c797d13b-d937-4080-95c9-c5b4df07eb35","Type":"ContainerStarted","Data":"f69c6acb662d6314cee6ac2a2b015a3b2cdadc672b1984cd24a9253c91ed97ee"}
Jul 10 18:49:40.227901 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:40.227875    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-xv7jv" event={"ID":"bdaecc62-345e-40da-96ef-c76e13c6d2e9","Type":"ContainerStarted","Data":"e52d245338f0b9adc1370dac3e026ea5977ae69d1ac35ba708ce8a4e632a0788"}
Jul 10 18:49:40.242580 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:40.242529    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-p4wg4" podStartSLOduration=1.722831606 podStartE2EDuration="4.242513824s" totalImagesPullingTime="2.519682218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:49:40.24120913 +0000 UTC m=+557.709127958" watchObservedRunningTime="2026-07-10 18:49:40.242513824 +0000 UTC m=+557.710432656"
Jul 10 18:49:40.256898 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:40.256834    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c89bb85-xv7jv" podStartSLOduration=1.631107778 podStartE2EDuration="4.256811986s" totalImagesPullingTime="2.625704208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:49:40.255821813 +0000 UTC m=+557.723740646" watchObservedRunningTime="2026-07-10 18:49:40.256811986 +0000 UTC m=+557.724730821"
Jul 10 18:49:40.275041 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:40.275007    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-xv7jv"]
Jul 10 18:49:42.235463 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:42.235423    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c89bb85-xv7jv" podUID="bdaecc62-345e-40da-96ef-c76e13c6d2e9" containerName="authorino" containerID="cri-o://e52d245338f0b9adc1370dac3e026ea5977ae69d1ac35ba708ce8a4e632a0788" gracePeriod=30
Jul 10 18:49:42.485399 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:42.485375    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-xv7jv"
Jul 10 18:49:42.566697 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:42.566664    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r676p\" (UniqueName: \"kubernetes.io/projected/bdaecc62-345e-40da-96ef-c76e13c6d2e9-kube-api-access-r676p\") pod \"bdaecc62-345e-40da-96ef-c76e13c6d2e9\" (UID: \"bdaecc62-345e-40da-96ef-c76e13c6d2e9\") "
Jul 10 18:49:42.568851 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:42.568828    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdaecc62-345e-40da-96ef-c76e13c6d2e9-kube-api-access-r676p" (OuterVolumeSpecName: "kube-api-access-r676p") pod "bdaecc62-345e-40da-96ef-c76e13c6d2e9" (UID: "bdaecc62-345e-40da-96ef-c76e13c6d2e9"). InnerVolumeSpecName "kube-api-access-r676p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:49:42.667541 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:42.667506    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r676p\" (UniqueName: \"kubernetes.io/projected/bdaecc62-345e-40da-96ef-c76e13c6d2e9-kube-api-access-r676p\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:49:43.240559 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:43.240522    2565 generic.go:356] "Generic (PLEG): container finished" podID="bdaecc62-345e-40da-96ef-c76e13c6d2e9" containerID="e52d245338f0b9adc1370dac3e026ea5977ae69d1ac35ba708ce8a4e632a0788" exitCode=0
Jul 10 18:49:43.240559 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:43.240560    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-xv7jv"
Jul 10 18:49:43.241037 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:43.240606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-xv7jv" event={"ID":"bdaecc62-345e-40da-96ef-c76e13c6d2e9","Type":"ContainerDied","Data":"e52d245338f0b9adc1370dac3e026ea5977ae69d1ac35ba708ce8a4e632a0788"}
Jul 10 18:49:43.241037 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:43.240643    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-xv7jv" event={"ID":"bdaecc62-345e-40da-96ef-c76e13c6d2e9","Type":"ContainerDied","Data":"0c17956ff1865fb8818021d98b7e058a445979eb3e9f7874d405a33efeef76dd"}
Jul 10 18:49:43.241037 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:43.240674    2565 scope.go:117] "RemoveContainer" containerID="e52d245338f0b9adc1370dac3e026ea5977ae69d1ac35ba708ce8a4e632a0788"
Jul 10 18:49:43.248864 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:43.248845    2565 scope.go:117] "RemoveContainer" containerID="e52d245338f0b9adc1370dac3e026ea5977ae69d1ac35ba708ce8a4e632a0788"
Jul 10 18:49:43.249160 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:49:43.249126    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52d245338f0b9adc1370dac3e026ea5977ae69d1ac35ba708ce8a4e632a0788\": container with ID starting with e52d245338f0b9adc1370dac3e026ea5977ae69d1ac35ba708ce8a4e632a0788 not found: ID does not exist" containerID="e52d245338f0b9adc1370dac3e026ea5977ae69d1ac35ba708ce8a4e632a0788"
Jul 10 18:49:43.249213 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:43.249169    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52d245338f0b9adc1370dac3e026ea5977ae69d1ac35ba708ce8a4e632a0788"} err="failed to get container status \"e52d245338f0b9adc1370dac3e026ea5977ae69d1ac35ba708ce8a4e632a0788\": rpc error: code = NotFound desc = could not find container \"e52d245338f0b9adc1370dac3e026ea5977ae69d1ac35ba708ce8a4e632a0788\": container with ID starting with e52d245338f0b9adc1370dac3e026ea5977ae69d1ac35ba708ce8a4e632a0788 not found: ID does not exist"
Jul 10 18:49:43.256387 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:43.256359    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-xv7jv"]
Jul 10 18:49:43.259784 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:43.259760    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-xv7jv"]
Jul 10 18:49:45.094060 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:49:45.094016    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdaecc62-345e-40da-96ef-c76e13c6d2e9" path="/var/lib/kubelet/pods/bdaecc62-345e-40da-96ef-c76e13c6d2e9/volumes"
Jul 10 18:50:01.054471 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.054439    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-pqfjx"]
Jul 10 18:50:01.054861 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.054712    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdaecc62-345e-40da-96ef-c76e13c6d2e9" containerName="authorino"
Jul 10 18:50:01.054861 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.054722    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdaecc62-345e-40da-96ef-c76e13c6d2e9" containerName="authorino"
Jul 10 18:50:01.054861 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.054780    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdaecc62-345e-40da-96ef-c76e13c6d2e9" containerName="authorino"
Jul 10 18:50:01.060781 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.060757    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-pqfjx"
Jul 10 18:50:01.064103 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.064072    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 10 18:50:01.066426 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.066396    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-pqfjx"]
Jul 10 18:50:01.212993 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.212946    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84b2d65b-d93f-4385-82d5-3e1df7ab3e97-tls-cert\") pod \"authorino-7c77957575-pqfjx\" (UID: \"84b2d65b-d93f-4385-82d5-3e1df7ab3e97\") " pod="kuadrant-system/authorino-7c77957575-pqfjx"
Jul 10 18:50:01.213222 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.213027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm57s\" (UniqueName: \"kubernetes.io/projected/84b2d65b-d93f-4385-82d5-3e1df7ab3e97-kube-api-access-vm57s\") pod \"authorino-7c77957575-pqfjx\" (UID: \"84b2d65b-d93f-4385-82d5-3e1df7ab3e97\") " pod="kuadrant-system/authorino-7c77957575-pqfjx"
Jul 10 18:50:01.314005 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.313902    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84b2d65b-d93f-4385-82d5-3e1df7ab3e97-tls-cert\") pod \"authorino-7c77957575-pqfjx\" (UID: \"84b2d65b-d93f-4385-82d5-3e1df7ab3e97\") " pod="kuadrant-system/authorino-7c77957575-pqfjx"
Jul 10 18:50:01.314005 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.313989    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm57s\" (UniqueName: \"kubernetes.io/projected/84b2d65b-d93f-4385-82d5-3e1df7ab3e97-kube-api-access-vm57s\") pod \"authorino-7c77957575-pqfjx\" (UID: \"84b2d65b-d93f-4385-82d5-3e1df7ab3e97\") " pod="kuadrant-system/authorino-7c77957575-pqfjx"
Jul 10 18:50:01.316494 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.316464    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84b2d65b-d93f-4385-82d5-3e1df7ab3e97-tls-cert\") pod \"authorino-7c77957575-pqfjx\" (UID: \"84b2d65b-d93f-4385-82d5-3e1df7ab3e97\") " pod="kuadrant-system/authorino-7c77957575-pqfjx"
Jul 10 18:50:01.326491 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.326463    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm57s\" (UniqueName: \"kubernetes.io/projected/84b2d65b-d93f-4385-82d5-3e1df7ab3e97-kube-api-access-vm57s\") pod \"authorino-7c77957575-pqfjx\" (UID: \"84b2d65b-d93f-4385-82d5-3e1df7ab3e97\") " pod="kuadrant-system/authorino-7c77957575-pqfjx"
Jul 10 18:50:01.371385 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.371340    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-pqfjx"
Jul 10 18:50:01.516583 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:01.516474    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-pqfjx"]
Jul 10 18:50:01.519134 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:50:01.519107    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84b2d65b_d93f_4385_82d5_3e1df7ab3e97.slice/crio-68d35618ab72277b49b17e2cfc5afc0dd96092f9bdf2f16ae14184cb60ddbd19 WatchSource:0}: Error finding container 68d35618ab72277b49b17e2cfc5afc0dd96092f9bdf2f16ae14184cb60ddbd19: Status 404 returned error can't find the container with id 68d35618ab72277b49b17e2cfc5afc0dd96092f9bdf2f16ae14184cb60ddbd19
Jul 10 18:50:02.320929 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:02.320831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-pqfjx" event={"ID":"84b2d65b-d93f-4385-82d5-3e1df7ab3e97","Type":"ContainerStarted","Data":"6b883eb0c5fa1a3f8d5530eb9e55273ac1b8760fc7da7e18f810dab6e5efc747"}
Jul 10 18:50:02.320929 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:02.320878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-pqfjx" event={"ID":"84b2d65b-d93f-4385-82d5-3e1df7ab3e97","Type":"ContainerStarted","Data":"68d35618ab72277b49b17e2cfc5afc0dd96092f9bdf2f16ae14184cb60ddbd19"}
Jul 10 18:50:02.343378 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:02.343323    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-pqfjx" podStartSLOduration=0.846338586 podStartE2EDuration="1.343306426s" totalImagesPullingTime="496.96784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:50:02.342451836 +0000 UTC m=+579.810370660" watchObservedRunningTime="2026-07-10 18:50:02.343306426 +0000 UTC m=+579.811225258"
Jul 10 18:50:02.373683 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:02.373643    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-p4wg4"]
Jul 10 18:50:02.373913 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:02.373875    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-p4wg4" podUID="c797d13b-d937-4080-95c9-c5b4df07eb35" containerName="authorino" containerID="cri-o://f69c6acb662d6314cee6ac2a2b015a3b2cdadc672b1984cd24a9253c91ed97ee" gracePeriod=30
Jul 10 18:50:02.630640 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:02.630609    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-p4wg4"
Jul 10 18:50:02.723835 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:02.723797    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q876f\" (UniqueName: \"kubernetes.io/projected/c797d13b-d937-4080-95c9-c5b4df07eb35-kube-api-access-q876f\") pod \"c797d13b-d937-4080-95c9-c5b4df07eb35\" (UID: \"c797d13b-d937-4080-95c9-c5b4df07eb35\") "
Jul 10 18:50:02.725882 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:02.725850    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c797d13b-d937-4080-95c9-c5b4df07eb35-kube-api-access-q876f" (OuterVolumeSpecName: "kube-api-access-q876f") pod "c797d13b-d937-4080-95c9-c5b4df07eb35" (UID: "c797d13b-d937-4080-95c9-c5b4df07eb35"). InnerVolumeSpecName "kube-api-access-q876f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:50:02.824746 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:02.824708    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q876f\" (UniqueName: \"kubernetes.io/projected/c797d13b-d937-4080-95c9-c5b4df07eb35-kube-api-access-q876f\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:50:03.326192 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:03.326158    2565 generic.go:356] "Generic (PLEG): container finished" podID="c797d13b-d937-4080-95c9-c5b4df07eb35" containerID="f69c6acb662d6314cee6ac2a2b015a3b2cdadc672b1984cd24a9253c91ed97ee" exitCode=0
Jul 10 18:50:03.326712 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:03.326201    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-p4wg4"
Jul 10 18:50:03.326712 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:03.326247    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-p4wg4" event={"ID":"c797d13b-d937-4080-95c9-c5b4df07eb35","Type":"ContainerDied","Data":"f69c6acb662d6314cee6ac2a2b015a3b2cdadc672b1984cd24a9253c91ed97ee"}
Jul 10 18:50:03.326712 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:03.326291    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-p4wg4" event={"ID":"c797d13b-d937-4080-95c9-c5b4df07eb35","Type":"ContainerDied","Data":"cf543493dae23cb0c7b9de3591631f03b2a363bfc35bcf0184b4648614033176"}
Jul 10 18:50:03.326712 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:03.326308    2565 scope.go:117] "RemoveContainer" containerID="f69c6acb662d6314cee6ac2a2b015a3b2cdadc672b1984cd24a9253c91ed97ee"
Jul 10 18:50:03.334685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:03.334668    2565 scope.go:117] "RemoveContainer" containerID="f69c6acb662d6314cee6ac2a2b015a3b2cdadc672b1984cd24a9253c91ed97ee"
Jul 10 18:50:03.334977 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:50:03.334958    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69c6acb662d6314cee6ac2a2b015a3b2cdadc672b1984cd24a9253c91ed97ee\": container with ID starting with f69c6acb662d6314cee6ac2a2b015a3b2cdadc672b1984cd24a9253c91ed97ee not found: ID does not exist" containerID="f69c6acb662d6314cee6ac2a2b015a3b2cdadc672b1984cd24a9253c91ed97ee"
Jul 10 18:50:03.335046 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:03.334987    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69c6acb662d6314cee6ac2a2b015a3b2cdadc672b1984cd24a9253c91ed97ee"} err="failed to get container status \"f69c6acb662d6314cee6ac2a2b015a3b2cdadc672b1984cd24a9253c91ed97ee\": rpc error: code = NotFound desc = could not find container \"f69c6acb662d6314cee6ac2a2b015a3b2cdadc672b1984cd24a9253c91ed97ee\": container with ID starting with f69c6acb662d6314cee6ac2a2b015a3b2cdadc672b1984cd24a9253c91ed97ee not found: ID does not exist"
Jul 10 18:50:03.343042 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:03.343015    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-p4wg4"]
Jul 10 18:50:03.346819 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:03.346791    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-p4wg4"]
Jul 10 18:50:05.093712 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:50:05.093681    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c797d13b-d937-4080-95c9-c5b4df07eb35" path="/var/lib/kubelet/pods/c797d13b-d937-4080-95c9-c5b4df07eb35/volumes"
Jul 10 18:51:49.112036 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.111995    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-89f8q"]
Jul 10 18:51:49.113411 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.112306    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c797d13b-d937-4080-95c9-c5b4df07eb35" containerName="authorino"
Jul 10 18:51:49.113411 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.112319    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c797d13b-d937-4080-95c9-c5b4df07eb35" containerName="authorino"
Jul 10 18:51:49.113411 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.112387    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c797d13b-d937-4080-95c9-c5b4df07eb35" containerName="authorino"
Jul 10 18:51:49.114010 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.113996    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-89f8q"
Jul 10 18:51:49.116974 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.116955    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 10 18:51:49.117105 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.116954    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 10 18:51:49.117105 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.117004    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 10 18:51:49.118297 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.118265    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-rkjvq\""
Jul 10 18:51:49.124546 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.124525    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-89f8q"]
Jul 10 18:51:49.178430 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.178393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chdmr\" (UniqueName: \"kubernetes.io/projected/c6097c37-d993-4954-a077-b8b6529a3c1b-kube-api-access-chdmr\") pod \"s3-init-89f8q\" (UID: \"c6097c37-d993-4954-a077-b8b6529a3c1b\") " pod="kserve/s3-init-89f8q"
Jul 10 18:51:49.279266 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.279231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chdmr\" (UniqueName: \"kubernetes.io/projected/c6097c37-d993-4954-a077-b8b6529a3c1b-kube-api-access-chdmr\") pod \"s3-init-89f8q\" (UID: \"c6097c37-d993-4954-a077-b8b6529a3c1b\") " pod="kserve/s3-init-89f8q"
Jul 10 18:51:49.288197 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.288165    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chdmr\" (UniqueName: \"kubernetes.io/projected/c6097c37-d993-4954-a077-b8b6529a3c1b-kube-api-access-chdmr\") pod \"s3-init-89f8q\" (UID: \"c6097c37-d993-4954-a077-b8b6529a3c1b\") " pod="kserve/s3-init-89f8q"
Jul 10 18:51:49.422731 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.422643    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-89f8q"
Jul 10 18:51:49.550189 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.550030    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-89f8q"]
Jul 10 18:51:49.552806 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:51:49.552777    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6097c37_d993_4954_a077_b8b6529a3c1b.slice/crio-b0552ca4745d6727c6eb93a916805cad34d04163ddcdca0b4736f66957381863 WatchSource:0}: Error finding container b0552ca4745d6727c6eb93a916805cad34d04163ddcdca0b4736f66957381863: Status 404 returned error can't find the container with id b0552ca4745d6727c6eb93a916805cad34d04163ddcdca0b4736f66957381863
Jul 10 18:51:49.783836 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:49.783793    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-89f8q" event={"ID":"c6097c37-d993-4954-a077-b8b6529a3c1b","Type":"ContainerStarted","Data":"b0552ca4745d6727c6eb93a916805cad34d04163ddcdca0b4736f66957381863"}
Jul 10 18:51:54.811121 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:54.811088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-89f8q" event={"ID":"c6097c37-d993-4954-a077-b8b6529a3c1b","Type":"ContainerStarted","Data":"1dba4dac88910439077ac582bfa2f5886f63706c6454c9e431e3e3ff05cc101d"}
Jul 10 18:51:54.829631 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:54.829571    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-89f8q" podStartSLOduration=1.180219876 podStartE2EDuration="5.829555351s" totalImagesPullingTime="4.649335475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:51:54.828198326 +0000 UTC m=+692.296117158" watchObservedRunningTime="2026-07-10 18:51:54.829555351 +0000 UTC m=+692.297474182"
Jul 10 18:51:57.826081 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:57.826046    2565 generic.go:356] "Generic (PLEG): container finished" podID="c6097c37-d993-4954-a077-b8b6529a3c1b" containerID="1dba4dac88910439077ac582bfa2f5886f63706c6454c9e431e3e3ff05cc101d" exitCode=0
Jul 10 18:51:57.826471 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:57.826116    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-89f8q" event={"ID":"c6097c37-d993-4954-a077-b8b6529a3c1b","Type":"ContainerDied","Data":"1dba4dac88910439077ac582bfa2f5886f63706c6454c9e431e3e3ff05cc101d"}
Jul 10 18:51:58.966214 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:58.966190    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-89f8q"
Jul 10 18:51:59.062434 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:59.062398    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chdmr\" (UniqueName: \"kubernetes.io/projected/c6097c37-d993-4954-a077-b8b6529a3c1b-kube-api-access-chdmr\") pod \"c6097c37-d993-4954-a077-b8b6529a3c1b\" (UID: \"c6097c37-d993-4954-a077-b8b6529a3c1b\") "
Jul 10 18:51:59.064604 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:59.064570    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6097c37-d993-4954-a077-b8b6529a3c1b-kube-api-access-chdmr" (OuterVolumeSpecName: "kube-api-access-chdmr") pod "c6097c37-d993-4954-a077-b8b6529a3c1b" (UID: "c6097c37-d993-4954-a077-b8b6529a3c1b"). InnerVolumeSpecName "kube-api-access-chdmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:51:59.163447 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:59.163416    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chdmr\" (UniqueName: \"kubernetes.io/projected/c6097c37-d993-4954-a077-b8b6529a3c1b-kube-api-access-chdmr\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:51:59.835110 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:59.835075    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-89f8q"
Jul 10 18:51:59.835279 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:59.835075    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-89f8q" event={"ID":"c6097c37-d993-4954-a077-b8b6529a3c1b","Type":"ContainerDied","Data":"b0552ca4745d6727c6eb93a916805cad34d04163ddcdca0b4736f66957381863"}
Jul 10 18:51:59.835279 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:51:59.835202    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0552ca4745d6727c6eb93a916805cad34d04163ddcdca0b4736f66957381863"
Jul 10 18:52:45.613591 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.613559    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"]
Jul 10 18:52:45.614072 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.613841    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6097c37-d993-4954-a077-b8b6529a3c1b" containerName="s3-init"
Jul 10 18:52:45.614072 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.613852    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6097c37-d993-4954-a077-b8b6529a3c1b" containerName="s3-init"
Jul 10 18:52:45.614072 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.613898    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6097c37-d993-4954-a077-b8b6529a3c1b" containerName="s3-init"
Jul 10 18:52:45.643657 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.643629    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"]
Jul 10 18:52:45.643813 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.643755    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.646832 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.646810    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 18:52:45.647906 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.647886    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jjdpb\""
Jul 10 18:52:45.648026 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.647886    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 10 18:52:45.648026 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.647894    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 18:52:45.734741 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.734708    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-dshm\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.734931 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.734756    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2e88754-b695-4188-8bb5-31312a3ebff3-tls-certs\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.734931 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.734804    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-model-cache\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.734931 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.734848    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.734931 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.734879    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-tmp-dir\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.734931 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.734904    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gs42\" (UniqueName: \"kubernetes.io/projected/b2e88754-b695-4188-8bb5-31312a3ebff3-kube-api-access-6gs42\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.735211 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.734993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-home\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.835637 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.835593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-home\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.835637 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.835646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-dshm\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.835901 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.835684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2e88754-b695-4188-8bb5-31312a3ebff3-tls-certs\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.835901 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.835702    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-model-cache\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.835901 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.835726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.835901 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.835752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-tmp-dir\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.835901 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.835799    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gs42\" (UniqueName: \"kubernetes.io/projected/b2e88754-b695-4188-8bb5-31312a3ebff3-kube-api-access-6gs42\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.836183 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.836086    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-home\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.836183 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.836116    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-model-cache\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.836265 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.836223    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.836302 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.836270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-tmp-dir\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.838040 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.838020    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-dshm\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.838342 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.838324    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2e88754-b695-4188-8bb5-31312a3ebff3-tls-certs\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.844856 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.844830    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gs42\" (UniqueName: \"kubernetes.io/projected/b2e88754-b695-4188-8bb5-31312a3ebff3-kube-api-access-6gs42\") pod \"scheduler-inline-config-test-kserve-f96d8754d-2tjs2\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:45.954418 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:45.954317    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:52:46.126751 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:46.126563    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"]
Jul 10 18:52:46.129660 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:52:46.129632    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e88754_b695_4188_8bb5_31312a3ebff3.slice/crio-8d4afc2f01d10b70cc48a1b39484cc140b91ae5f870715f3ac62fe0c3d885888 WatchSource:0}: Error finding container 8d4afc2f01d10b70cc48a1b39484cc140b91ae5f870715f3ac62fe0c3d885888: Status 404 returned error can't find the container with id 8d4afc2f01d10b70cc48a1b39484cc140b91ae5f870715f3ac62fe0c3d885888
Jul 10 18:52:46.131793 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:46.131771    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:52:47.037508 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:47.037472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2" event={"ID":"b2e88754-b695-4188-8bb5-31312a3ebff3","Type":"ContainerStarted","Data":"8d4afc2f01d10b70cc48a1b39484cc140b91ae5f870715f3ac62fe0c3d885888"}
Jul 10 18:52:50.054046 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:50.054008    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2" event={"ID":"b2e88754-b695-4188-8bb5-31312a3ebff3","Type":"ContainerStarted","Data":"791239b9b90c3f87bcf6da79fff1e30a8140a8d48b82bcfd9b2868705271966f"}
Jul 10 18:52:54.072064 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:54.072026    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2e88754-b695-4188-8bb5-31312a3ebff3" containerID="791239b9b90c3f87bcf6da79fff1e30a8140a8d48b82bcfd9b2868705271966f" exitCode=0
Jul 10 18:52:54.072511 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:54.072108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2" event={"ID":"b2e88754-b695-4188-8bb5-31312a3ebff3","Type":"ContainerDied","Data":"791239b9b90c3f87bcf6da79fff1e30a8140a8d48b82bcfd9b2868705271966f"}
Jul 10 18:52:56.083304 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:56.083263    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2" event={"ID":"b2e88754-b695-4188-8bb5-31312a3ebff3","Type":"ContainerStarted","Data":"85b39932ba7b4b9d2783a904f0d150f1e2c0f7a4b846346ac813f5772f8ae972"}
Jul 10 18:52:56.104740 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:52:56.104687    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2" podStartSLOduration=2.263262539 podStartE2EDuration="11.104671153s" totalImagesPullingTime="4.443744727s" totalInitContainerRuntime="4.397663887s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:52:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:52:56.103375039 +0000 UTC m=+753.571293875" watchObservedRunningTime="2026-07-10 18:52:56.104671153 +0000 UTC m=+753.572589984"
Jul 10 18:53:05.955323 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:05.955281    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:53:05.955728 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:05.955352    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:53:05.968199 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:05.968169    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:53:06.134385 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:06.134353    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:53:38.044203 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.044171    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"]
Jul 10 18:53:38.044750 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.044402    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2" podUID="b2e88754-b695-4188-8bb5-31312a3ebff3" containerName="main" containerID="cri-o://85b39932ba7b4b9d2783a904f0d150f1e2c0f7a4b846346ac813f5772f8ae972" gracePeriod=30
Jul 10 18:53:38.257653 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.257619    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2e88754-b695-4188-8bb5-31312a3ebff3" containerID="85b39932ba7b4b9d2783a904f0d150f1e2c0f7a4b846346ac813f5772f8ae972" exitCode=0
Jul 10 18:53:38.257827 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.257683    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2" event={"ID":"b2e88754-b695-4188-8bb5-31312a3ebff3","Type":"ContainerDied","Data":"85b39932ba7b4b9d2783a904f0d150f1e2c0f7a4b846346ac813f5772f8ae972"}
Jul 10 18:53:38.286434 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.286416    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:53:38.367255 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.367191    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2e88754-b695-4188-8bb5-31312a3ebff3-tls-certs\") pod \"b2e88754-b695-4188-8bb5-31312a3ebff3\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") "
Jul 10 18:53:38.367255 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.367232    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-dshm\") pod \"b2e88754-b695-4188-8bb5-31312a3ebff3\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") "
Jul 10 18:53:38.367449 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.367274    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-model-cache\") pod \"b2e88754-b695-4188-8bb5-31312a3ebff3\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") "
Jul 10 18:53:38.367449 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.367299    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-home\") pod \"b2e88754-b695-4188-8bb5-31312a3ebff3\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") "
Jul 10 18:53:38.367449 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.367332    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gs42\" (UniqueName: \"kubernetes.io/projected/b2e88754-b695-4188-8bb5-31312a3ebff3-kube-api-access-6gs42\") pod \"b2e88754-b695-4188-8bb5-31312a3ebff3\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") "
Jul 10 18:53:38.367449 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.367374    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-kserve-provision-location\") pod \"b2e88754-b695-4188-8bb5-31312a3ebff3\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") "
Jul 10 18:53:38.367449 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.367401    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-tmp-dir\") pod \"b2e88754-b695-4188-8bb5-31312a3ebff3\" (UID: \"b2e88754-b695-4188-8bb5-31312a3ebff3\") "
Jul 10 18:53:38.367702 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.367563    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2e88754-b695-4188-8bb5-31312a3ebff3" (UID: "b2e88754-b695-4188-8bb5-31312a3ebff3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:53:38.367702 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.367589    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-home" (OuterVolumeSpecName: "home") pod "b2e88754-b695-4188-8bb5-31312a3ebff3" (UID: "b2e88754-b695-4188-8bb5-31312a3ebff3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:53:38.367702 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.367657    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:53:38.367702 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.367687    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:53:38.367897 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.367752    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2e88754-b695-4188-8bb5-31312a3ebff3" (UID: "b2e88754-b695-4188-8bb5-31312a3ebff3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:53:38.369618 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.369589    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-dshm" (OuterVolumeSpecName: "dshm") pod "b2e88754-b695-4188-8bb5-31312a3ebff3" (UID: "b2e88754-b695-4188-8bb5-31312a3ebff3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:53:38.369728 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.369625    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2e88754-b695-4188-8bb5-31312a3ebff3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2e88754-b695-4188-8bb5-31312a3ebff3" (UID: "b2e88754-b695-4188-8bb5-31312a3ebff3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:53:38.369774 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.369728    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e88754-b695-4188-8bb5-31312a3ebff3-kube-api-access-6gs42" (OuterVolumeSpecName: "kube-api-access-6gs42") pod "b2e88754-b695-4188-8bb5-31312a3ebff3" (UID: "b2e88754-b695-4188-8bb5-31312a3ebff3"). InnerVolumeSpecName "kube-api-access-6gs42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:53:38.423518 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.423484    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b2e88754-b695-4188-8bb5-31312a3ebff3" (UID: "b2e88754-b695-4188-8bb5-31312a3ebff3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:53:38.468974 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.468947    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2e88754-b695-4188-8bb5-31312a3ebff3-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:53:38.468974 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.468971    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:53:38.468974 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.468981    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gs42\" (UniqueName: \"kubernetes.io/projected/b2e88754-b695-4188-8bb5-31312a3ebff3-kube-api-access-6gs42\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:53:38.469195 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.468990    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:53:38.469195 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:38.468998    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2e88754-b695-4188-8bb5-31312a3ebff3-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:53:39.264333 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:39.264307    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"
Jul 10 18:53:39.264719 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:39.264306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2" event={"ID":"b2e88754-b695-4188-8bb5-31312a3ebff3","Type":"ContainerDied","Data":"8d4afc2f01d10b70cc48a1b39484cc140b91ae5f870715f3ac62fe0c3d885888"}
Jul 10 18:53:39.264719 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:39.264444    2565 scope.go:117] "RemoveContainer" containerID="85b39932ba7b4b9d2783a904f0d150f1e2c0f7a4b846346ac813f5772f8ae972"
Jul 10 18:53:39.272396 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:39.272379    2565 scope.go:117] "RemoveContainer" containerID="791239b9b90c3f87bcf6da79fff1e30a8140a8d48b82bcfd9b2868705271966f"
Jul 10 18:53:39.283501 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:39.283475    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"]
Jul 10 18:53:39.288102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:39.288082    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f96d8754d-2tjs2"]
Jul 10 18:53:41.093710 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.093677    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2e88754-b695-4188-8bb5-31312a3ebff3" path="/var/lib/kubelet/pods/b2e88754-b695-4188-8bb5-31312a3ebff3/volumes"
Jul 10 18:53:41.825323 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.825289    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"]
Jul 10 18:53:41.825645 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.825627    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2e88754-b695-4188-8bb5-31312a3ebff3" containerName="storage-initializer"
Jul 10 18:53:41.825728 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.825647    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2e88754-b695-4188-8bb5-31312a3ebff3" containerName="storage-initializer"
Jul 10 18:53:41.825728 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.825670    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2e88754-b695-4188-8bb5-31312a3ebff3" containerName="main"
Jul 10 18:53:41.825728 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.825678    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2e88754-b695-4188-8bb5-31312a3ebff3" containerName="main"
Jul 10 18:53:41.825876 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.825749    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2e88754-b695-4188-8bb5-31312a3ebff3" containerName="main"
Jul 10 18:53:41.829120 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.829098    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:41.831946 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.831928    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jjdpb\""
Jul 10 18:53:41.833556 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.833538    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 18:53:41.833704 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.833519    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 18:53:41.833905 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.833744    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 10 18:53:41.835621 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.835601    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"]
Jul 10 18:53:41.900206 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.900171    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-dshm\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:41.900206 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.900204    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:41.900441 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.900298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-model-cache\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:41.900441 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.900352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:41.900441 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.900379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:41.900441 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.900402    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jnhj\" (UniqueName: \"kubernetes.io/projected/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-kube-api-access-9jnhj\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:41.900598 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:41.900443    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-home\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.001526 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.001491    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-dshm\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.001526 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.001523    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.001745 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.001583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-model-cache\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.001745 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.001609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.001745 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.001700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.001745 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.001739    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jnhj\" (UniqueName: \"kubernetes.io/projected/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-kube-api-access-9jnhj\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.001938 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.001784    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-home\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.002083 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.002053    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-model-cache\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.002142 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.002088    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.002227 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.002167    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.002227 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.002208    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-home\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.003884 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.003861    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-dshm\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.004125 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.004108    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.011357 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.011335    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jnhj\" (UniqueName: \"kubernetes.io/projected/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-kube-api-access-9jnhj\") pod \"scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.140889 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.140796    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:42.267454 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.267422    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"]
Jul 10 18:53:42.269574 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:53:42.269540    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69115b2_ef7b_4de2_9f4c_d1c64c7b8aa1.slice/crio-cd84569b60bdc88ef677548216269233f4e8dfea6128039bfd24a352416534d1 WatchSource:0}: Error finding container cd84569b60bdc88ef677548216269233f4e8dfea6128039bfd24a352416534d1: Status 404 returned error can't find the container with id cd84569b60bdc88ef677548216269233f4e8dfea6128039bfd24a352416534d1
Jul 10 18:53:42.278925 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:42.278898    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw" event={"ID":"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1","Type":"ContainerStarted","Data":"cd84569b60bdc88ef677548216269233f4e8dfea6128039bfd24a352416534d1"}
Jul 10 18:53:43.285864 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:43.285817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw" event={"ID":"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1","Type":"ContainerStarted","Data":"6be50e3178207db54e93192ffef6f311d3a2e8b934804cca01074c97206df7eb"}
Jul 10 18:53:47.304977 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:47.304939    2565 generic.go:356] "Generic (PLEG): container finished" podID="f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" containerID="6be50e3178207db54e93192ffef6f311d3a2e8b934804cca01074c97206df7eb" exitCode=0
Jul 10 18:53:47.305428 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:47.305006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw" event={"ID":"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1","Type":"ContainerDied","Data":"6be50e3178207db54e93192ffef6f311d3a2e8b934804cca01074c97206df7eb"}
Jul 10 18:53:48.310667 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:48.310628    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw" event={"ID":"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1","Type":"ContainerStarted","Data":"44ce4156976cdffeea4cb1df7b56b717edbfb176d98096c613a606a5c003333d"}
Jul 10 18:53:48.332706 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:48.332664    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw" podStartSLOduration=3.215625903 podStartE2EDuration="7.33264892s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.117023017s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:53:48.332185547 +0000 UTC m=+805.800104402" watchObservedRunningTime="2026-07-10 18:53:48.33264892 +0000 UTC m=+805.800567751"
Jul 10 18:53:52.141712 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:52.141680    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:52.142119 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:52.141755    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:52.154136 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:52.154116    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:53:52.338280 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:53:52.338254    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:54:15.257804 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.257770    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"]
Jul 10 18:54:15.258372 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.257991    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw" podUID="f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" containerName="main" containerID="cri-o://44ce4156976cdffeea4cb1df7b56b717edbfb176d98096c613a606a5c003333d" gracePeriod=30
Jul 10 18:54:15.427607 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.427573    2565 generic.go:356] "Generic (PLEG): container finished" podID="f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" containerID="44ce4156976cdffeea4cb1df7b56b717edbfb176d98096c613a606a5c003333d" exitCode=0
Jul 10 18:54:15.427735 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.427641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw" event={"ID":"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1","Type":"ContainerDied","Data":"44ce4156976cdffeea4cb1df7b56b717edbfb176d98096c613a606a5c003333d"}
Jul 10 18:54:15.505766 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.505735    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:54:15.565937 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.565873    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-tmp-dir\") pod \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") "
Jul 10 18:54:15.565937 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.565910    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jnhj\" (UniqueName: \"kubernetes.io/projected/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-kube-api-access-9jnhj\") pod \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") "
Jul 10 18:54:15.565937 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.565937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-model-cache\") pod \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") "
Jul 10 18:54:15.566191 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.565953    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-kserve-provision-location\") pod \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") "
Jul 10 18:54:15.566191 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.565983    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-tls-certs\") pod \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") "
Jul 10 18:54:15.566191 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.566010    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-dshm\") pod \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") "
Jul 10 18:54:15.566191 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.566046    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-home\") pod \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\" (UID: \"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1\") "
Jul 10 18:54:15.566191 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.566132    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" (UID: "f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:54:15.566446 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.566187    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-model-cache" (OuterVolumeSpecName: "model-cache") pod "f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" (UID: "f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:54:15.566446 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.566331    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-home" (OuterVolumeSpecName: "home") pod "f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" (UID: "f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:54:15.566446 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.566342    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:54:15.566446 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.566359    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:54:15.568006 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.567980    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-kube-api-access-9jnhj" (OuterVolumeSpecName: "kube-api-access-9jnhj") pod "f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" (UID: "f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1"). InnerVolumeSpecName "kube-api-access-9jnhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:54:15.568543 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.568523    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" (UID: "f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:54:15.568641 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.568621    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-dshm" (OuterVolumeSpecName: "dshm") pod "f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" (UID: "f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:54:15.620959 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.620932    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" (UID: "f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:54:15.667589 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.667556    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jnhj\" (UniqueName: \"kubernetes.io/projected/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-kube-api-access-9jnhj\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:54:15.667589 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.667584    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:54:15.667765 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.667599    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:54:15.667765 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.667610    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:54:15.667765 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:15.667617    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:54:16.433794 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:16.433763    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"
Jul 10 18:54:16.433794 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:16.433769    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw" event={"ID":"f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1","Type":"ContainerDied","Data":"cd84569b60bdc88ef677548216269233f4e8dfea6128039bfd24a352416534d1"}
Jul 10 18:54:16.434313 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:16.433837    2565 scope.go:117] "RemoveContainer" containerID="44ce4156976cdffeea4cb1df7b56b717edbfb176d98096c613a606a5c003333d"
Jul 10 18:54:16.442102 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:16.442083    2565 scope.go:117] "RemoveContainer" containerID="6be50e3178207db54e93192ffef6f311d3a2e8b934804cca01074c97206df7eb"
Jul 10 18:54:16.457033 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:16.456999    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"]
Jul 10 18:54:16.461433 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:16.461410    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-6555bfb5c6-przlw"]
Jul 10 18:54:17.094784 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:17.094750    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" path="/var/lib/kubelet/pods/f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1/volumes"
Jul 10 18:54:27.858765 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.858734    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"]
Jul 10 18:54:27.859140 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.859008    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" containerName="main"
Jul 10 18:54:27.859140 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.859018    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" containerName="main"
Jul 10 18:54:27.859140 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.859032    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" containerName="storage-initializer"
Jul 10 18:54:27.859140 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.859038    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" containerName="storage-initializer"
Jul 10 18:54:27.859140 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.859089    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f69115b2-ef7b-4de2-9f4c-d1c64c7b8aa1" containerName="main"
Jul 10 18:54:27.861677 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.861660    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:27.871841 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.871818    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 18:54:27.871841 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.871819    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 10 18:54:27.872036 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.871958    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jjdpb\""
Jul 10 18:54:27.872328 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.872314    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 18:54:27.893577 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.893553    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"]
Jul 10 18:54:27.963174 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.963115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:27.963174 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.963176    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-model-cache\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:27.963423 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.963272    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp5ks\" (UniqueName: \"kubernetes.io/projected/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-kube-api-access-wp5ks\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:27.963423 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.963323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:27.963423 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.963400    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-home\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:27.963540 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.963434    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-dshm\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:27.963540 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:27.963470    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.063940 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.063901    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.063940 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.063939    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-model-cache\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.064215 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.063996    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp5ks\" (UniqueName: \"kubernetes.io/projected/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-kube-api-access-wp5ks\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.064215 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.064032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.064215 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.064068    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-home\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.064215 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.064092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-dshm\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.064215 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.064134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.064490 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.064417    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.064490 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.064468    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-home\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.064565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.064517    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.064565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.064519    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-model-cache\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.066362 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.066343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-dshm\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.066834 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.066813    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.073093 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.073070    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp5ks\" (UniqueName: \"kubernetes.io/projected/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-kube-api-access-wp5ks\") pod \"scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.171517 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.171438    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:28.248773 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.248741    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"]
Jul 10 18:54:28.251565 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.251545    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.256428 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.256308    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-kq4vp\""
Jul 10 18:54:28.271228 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.271179    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"]
Jul 10 18:54:28.332236 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.332203    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"]
Jul 10 18:54:28.335339 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:54:28.335297    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c7b7e3_2c6a_4c5f_80fa_4fd76afd8e4c.slice/crio-b8c1303e69ce1047ad79afb5f1f19520125a3eb2f4c5e186d9d5f7855ab5d447 WatchSource:0}: Error finding container b8c1303e69ce1047ad79afb5f1f19520125a3eb2f4c5e186d9d5f7855ab5d447: Status 404 returned error can't find the container with id b8c1303e69ce1047ad79afb5f1f19520125a3eb2f4c5e186d9d5f7855ab5d447
Jul 10 18:54:28.366739 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.366716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.366855 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.366746    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.366855 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.366766    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dhlm\" (UniqueName: \"kubernetes.io/projected/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-kube-api-access-2dhlm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.366855 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.366789    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.367014 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.366871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.367014 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.366987    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.468485 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.468385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.468485 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.468442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.468485 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.468470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dhlm\" (UniqueName: \"kubernetes.io/projected/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-kube-api-access-2dhlm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.468759 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.468497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.468759 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.468561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.468759 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.468691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.469009 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.468838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.469009 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.468989    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.469140 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.469080    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.469140 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.469133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.471136 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.471120    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.480012 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.479977    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dhlm\" (UniqueName: \"kubernetes.io/projected/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-kube-api-access-2dhlm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.487653 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.487621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm" event={"ID":"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c","Type":"ContainerStarted","Data":"00d48578b62b91fab6ee5095ea8d539892ad2b5efc0ae2f770f34691a7073c34"}
Jul 10 18:54:28.487744 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.487680    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm" event={"ID":"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c","Type":"ContainerStarted","Data":"b8c1303e69ce1047ad79afb5f1f19520125a3eb2f4c5e186d9d5f7855ab5d447"}
Jul 10 18:54:28.564085 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.564049    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:54:28.722014 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:28.721993    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"]
Jul 10 18:54:28.724930 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:54:28.724903    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb846fb8_bd40_4c24_8d93_09cacea6d6e1.slice/crio-b5a347c0ba4efdc20fa64b83cf87569eb07dbe2ed8f55f83c25d687795989068 WatchSource:0}: Error finding container b5a347c0ba4efdc20fa64b83cf87569eb07dbe2ed8f55f83c25d687795989068: Status 404 returned error can't find the container with id b5a347c0ba4efdc20fa64b83cf87569eb07dbe2ed8f55f83c25d687795989068
Jul 10 18:54:29.494844 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:29.494812    2565 generic.go:356] "Generic (PLEG): container finished" podID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerID="ee1f2732ad209343dc4b2b5c01982f4813a95104214abcc99ef5aff7b6e68590" exitCode=0
Jul 10 18:54:29.495268 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:29.494885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" event={"ID":"bb846fb8-bd40-4c24-8d93-09cacea6d6e1","Type":"ContainerDied","Data":"ee1f2732ad209343dc4b2b5c01982f4813a95104214abcc99ef5aff7b6e68590"}
Jul 10 18:54:29.495268 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:29.494936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" event={"ID":"bb846fb8-bd40-4c24-8d93-09cacea6d6e1","Type":"ContainerStarted","Data":"b5a347c0ba4efdc20fa64b83cf87569eb07dbe2ed8f55f83c25d687795989068"}
Jul 10 18:54:30.503593 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:30.503560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" event={"ID":"bb846fb8-bd40-4c24-8d93-09cacea6d6e1","Type":"ContainerStarted","Data":"dfc0138753b936f722d951923dfc67475746d1663b1456bdb2f67b17673f4189"}
Jul 10 18:54:33.520197 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:33.520127    2565 generic.go:356] "Generic (PLEG): container finished" podID="59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" containerID="00d48578b62b91fab6ee5095ea8d539892ad2b5efc0ae2f770f34691a7073c34" exitCode=0
Jul 10 18:54:33.520197 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:33.520186    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm" event={"ID":"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c","Type":"ContainerDied","Data":"00d48578b62b91fab6ee5095ea8d539892ad2b5efc0ae2f770f34691a7073c34"}
Jul 10 18:54:34.526905 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:34.526859    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm" event={"ID":"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c","Type":"ContainerStarted","Data":"282bb1b8b6ba2c8a57b8e920832ed8f6408c5049b174cdb46ad812c36f290d63"}
Jul 10 18:54:34.552645 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:34.552587    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm" podStartSLOduration=2.880138204 podStartE2EDuration="7.552568557s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.672430353s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:54:34.551904122 +0000 UTC m=+852.019822981" watchObservedRunningTime="2026-07-10 18:54:34.552568557 +0000 UTC m=+852.020487389"
Jul 10 18:54:38.171900 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:38.171861    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:38.172463 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:38.171942    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:38.187229 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:38.187201    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:54:38.559385 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:54:38.559356    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:55:00.655578 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:00.655540    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" event={"ID":"bb846fb8-bd40-4c24-8d93-09cacea6d6e1","Type":"ContainerStarted","Data":"d05ce77b681eb40e036ac48a86c7a1d4de5a44b8586a6ecd10555af0d1ba10a2"}
Jul 10 18:55:00.655969 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:00.655754    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:55:00.658550 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:00.658510    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 10 18:55:00.679344 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:00.679280    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" podStartSLOduration=1.044242165 podStartE2EDuration="32.679262311s" totalImagesPullingTime="30.949764031s" totalInitContainerRuntime="685.256115ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:54:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:55:00.676875008 +0000 UTC m=+878.144793868" watchObservedRunningTime="2026-07-10 18:55:00.679262311 +0000 UTC m=+878.147181144"
Jul 10 18:55:01.662223 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:01.662179    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 10 18:55:02.719118 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.719081    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"]
Jul 10 18:55:02.719636 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.719451    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm" podUID="59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" containerName="main" containerID="cri-o://282bb1b8b6ba2c8a57b8e920832ed8f6408c5049b174cdb46ad812c36f290d63" gracePeriod=30
Jul 10 18:55:02.723409 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.723371    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"]
Jul 10 18:55:02.723702 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.723671    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="main" containerID="cri-o://dfc0138753b936f722d951923dfc67475746d1663b1456bdb2f67b17673f4189" gracePeriod=30
Jul 10 18:55:02.723924 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.723886    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="tokenizer" containerID="cri-o://d05ce77b681eb40e036ac48a86c7a1d4de5a44b8586a6ecd10555af0d1ba10a2" gracePeriod=30
Jul 10 18:55:02.725888 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.725838    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 10 18:55:02.970166 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.970082    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:55:02.989156 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.989113    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-kserve-provision-location\") pod \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") "
Jul 10 18:55:02.989309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.989224    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-model-cache\") pod \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") "
Jul 10 18:55:02.989309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.989257    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-tmp-dir\") pod \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") "
Jul 10 18:55:02.989309 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.989279    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-dshm\") pod \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") "
Jul 10 18:55:02.989481 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.989328    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-home\") pod \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") "
Jul 10 18:55:02.989481 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.989357    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-tls-certs\") pod \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") "
Jul 10 18:55:02.989481 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.989395    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp5ks\" (UniqueName: \"kubernetes.io/projected/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-kube-api-access-wp5ks\") pod \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\" (UID: \"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c\") "
Jul 10 18:55:02.989627 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.989491    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" (UID: "59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:02.989627 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.989519    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" (UID: "59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:02.989720 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.989656    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:02.989720 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.989676    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:02.989835 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.989772    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-home" (OuterVolumeSpecName: "home") pod "59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" (UID: "59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:02.992141 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.992103    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" (UID: "59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:55:02.992141 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.992124    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-kube-api-access-wp5ks" (OuterVolumeSpecName: "kube-api-access-wp5ks") pod "59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" (UID: "59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c"). InnerVolumeSpecName "kube-api-access-wp5ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:55:02.992362 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:02.992323    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-dshm" (OuterVolumeSpecName: "dshm") pod "59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" (UID: "59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:03.059024 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.058945    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" (UID: "59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:03.090916 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.090876    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:03.090916 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.090916    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:03.091096 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.090932    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:03.091096 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.090945    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:03.091096 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.090961    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp5ks\" (UniqueName: \"kubernetes.io/projected/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c-kube-api-access-wp5ks\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:03.671269 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.671234    2565 generic.go:356] "Generic (PLEG): container finished" podID="59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" containerID="282bb1b8b6ba2c8a57b8e920832ed8f6408c5049b174cdb46ad812c36f290d63" exitCode=0
Jul 10 18:55:03.671469 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.671295    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"
Jul 10 18:55:03.671469 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.671331    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm" event={"ID":"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c","Type":"ContainerDied","Data":"282bb1b8b6ba2c8a57b8e920832ed8f6408c5049b174cdb46ad812c36f290d63"}
Jul 10 18:55:03.671469 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.671397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm" event={"ID":"59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c","Type":"ContainerDied","Data":"b8c1303e69ce1047ad79afb5f1f19520125a3eb2f4c5e186d9d5f7855ab5d447"}
Jul 10 18:55:03.671469 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.671428    2565 scope.go:117] "RemoveContainer" containerID="282bb1b8b6ba2c8a57b8e920832ed8f6408c5049b174cdb46ad812c36f290d63"
Jul 10 18:55:03.673978 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.673946    2565 generic.go:356] "Generic (PLEG): container finished" podID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerID="dfc0138753b936f722d951923dfc67475746d1663b1456bdb2f67b17673f4189" exitCode=0
Jul 10 18:55:03.674097 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.674051    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" event={"ID":"bb846fb8-bd40-4c24-8d93-09cacea6d6e1","Type":"ContainerDied","Data":"dfc0138753b936f722d951923dfc67475746d1663b1456bdb2f67b17673f4189"}
Jul 10 18:55:03.680235 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.680211    2565 scope.go:117] "RemoveContainer" containerID="00d48578b62b91fab6ee5095ea8d539892ad2b5efc0ae2f770f34691a7073c34"
Jul 10 18:55:03.692654 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.692620    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"]
Jul 10 18:55:03.698163 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.698119    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-d9c8d8748-sz5cm"]
Jul 10 18:55:03.744580 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.744385    2565 scope.go:117] "RemoveContainer" containerID="282bb1b8b6ba2c8a57b8e920832ed8f6408c5049b174cdb46ad812c36f290d63"
Jul 10 18:55:03.744816 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:55:03.744771    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"282bb1b8b6ba2c8a57b8e920832ed8f6408c5049b174cdb46ad812c36f290d63\": container with ID starting with 282bb1b8b6ba2c8a57b8e920832ed8f6408c5049b174cdb46ad812c36f290d63 not found: ID does not exist" containerID="282bb1b8b6ba2c8a57b8e920832ed8f6408c5049b174cdb46ad812c36f290d63"
Jul 10 18:55:03.744816 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.744801    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282bb1b8b6ba2c8a57b8e920832ed8f6408c5049b174cdb46ad812c36f290d63"} err="failed to get container status \"282bb1b8b6ba2c8a57b8e920832ed8f6408c5049b174cdb46ad812c36f290d63\": rpc error: code = NotFound desc = could not find container \"282bb1b8b6ba2c8a57b8e920832ed8f6408c5049b174cdb46ad812c36f290d63\": container with ID starting with 282bb1b8b6ba2c8a57b8e920832ed8f6408c5049b174cdb46ad812c36f290d63 not found: ID does not exist"
Jul 10 18:55:03.744894 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.744821    2565 scope.go:117] "RemoveContainer" containerID="00d48578b62b91fab6ee5095ea8d539892ad2b5efc0ae2f770f34691a7073c34"
Jul 10 18:55:03.745099 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:55:03.745083    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d48578b62b91fab6ee5095ea8d539892ad2b5efc0ae2f770f34691a7073c34\": container with ID starting with 00d48578b62b91fab6ee5095ea8d539892ad2b5efc0ae2f770f34691a7073c34 not found: ID does not exist" containerID="00d48578b62b91fab6ee5095ea8d539892ad2b5efc0ae2f770f34691a7073c34"
Jul 10 18:55:03.745178 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:03.745102    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d48578b62b91fab6ee5095ea8d539892ad2b5efc0ae2f770f34691a7073c34"} err="failed to get container status \"00d48578b62b91fab6ee5095ea8d539892ad2b5efc0ae2f770f34691a7073c34\": rpc error: code = NotFound desc = could not find container \"00d48578b62b91fab6ee5095ea8d539892ad2b5efc0ae2f770f34691a7073c34\": container with ID starting with 00d48578b62b91fab6ee5095ea8d539892ad2b5efc0ae2f770f34691a7073c34 not found: ID does not exist"
Jul 10 18:55:05.094242 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:05.094205    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" path="/var/lib/kubelet/pods/59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c/volumes"
Jul 10 18:55:08.564372 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:08.564325    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:55:12.724360 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:55:12.724331    2565 logging.go:55] [core] [Channel #26 SubChannel #27]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.28:9003", ServerName: "10.134.0.28:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.28:9003: connect: connection refused"
Jul 10 18:55:13.724791 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:13.724744    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.28:9003\" within 1s: context deadline exceeded"
Jul 10 18:55:19.946779 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:19.946698    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"]
Jul 10 18:55:19.947268 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:19.947013    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" containerName="main"
Jul 10 18:55:19.947268 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:19.947025    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" containerName="main"
Jul 10 18:55:19.947268 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:19.947035    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" containerName="storage-initializer"
Jul 10 18:55:19.947268 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:19.947040    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" containerName="storage-initializer"
Jul 10 18:55:19.947268 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:19.947103    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c7b7e3-2c6a-4c5f-80fa-4fd76afd8e4c" containerName="main"
Jul 10 18:55:19.948946 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:19.948925    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:19.951786 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:19.951765    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 10 18:55:19.962926 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:19.962899    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"]
Jul 10 18:55:20.039747 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.039712    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.039910 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.039756    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-home\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.039910 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.039781    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-dshm\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.039910 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.039840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-model-cache\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.039910 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.039877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gslgd\" (UniqueName: \"kubernetes.io/projected/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-kube-api-access-gslgd\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.040043 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.039911    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.040043 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.039990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-tls-certs\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.140862 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.140828    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-dshm\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.140862 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.140865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-model-cache\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.141120 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.140886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gslgd\" (UniqueName: \"kubernetes.io/projected/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-kube-api-access-gslgd\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.141120 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.140917    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.141120 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.140954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-tls-certs\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.141120 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.140982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.141120 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.141013    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-home\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.141400 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.141365    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-model-cache\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.141400 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.141385    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.141479 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.141471    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-home\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.141512 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.141470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.143421 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.143390    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-dshm\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.143535 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.143519    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-tls-certs\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.153298 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.153265    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gslgd\" (UniqueName: \"kubernetes.io/projected/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-kube-api-access-gslgd\") pod \"precise-prefix-cache-test-kserve-6657b58549-hk5t7\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.279118 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.279083    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:20.417890 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.417715    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"]
Jul 10 18:55:20.420416 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:55:20.420386    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode295dc0a_1c37_4fe6_bc4d_b41159a2d499.slice/crio-e6bc33565aa7cd41d63d40d3a0f116f8debb6eb661e0780d37800a0a6d4e2068 WatchSource:0}: Error finding container e6bc33565aa7cd41d63d40d3a0f116f8debb6eb661e0780d37800a0a6d4e2068: Status 404 returned error can't find the container with id e6bc33565aa7cd41d63d40d3a0f116f8debb6eb661e0780d37800a0a6d4e2068
Jul 10 18:55:20.756213 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.756178    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7" event={"ID":"e295dc0a-1c37-4fe6-bc4d-b41159a2d499","Type":"ContainerStarted","Data":"df45616bb2f2b8038baaeda2c377e17ad320350a7daeedcc00faf4515dc2b4e7"}
Jul 10 18:55:20.756406 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:20.756269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7" event={"ID":"e295dc0a-1c37-4fe6-bc4d-b41159a2d499","Type":"ContainerStarted","Data":"e6bc33565aa7cd41d63d40d3a0f116f8debb6eb661e0780d37800a0a6d4e2068"}
Jul 10 18:55:22.725196 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:55:22.725136    2565 logging.go:55] [core] [Channel #28 SubChannel #29]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.28:9003", ServerName: "10.134.0.28:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.28:9003: connect: connection refused"
Jul 10 18:55:23.725593 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:23.725549    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.28:9003\" within 1s: context deadline exceeded"
Jul 10 18:55:25.779909 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:25.779876    2565 generic.go:356] "Generic (PLEG): container finished" podID="e295dc0a-1c37-4fe6-bc4d-b41159a2d499" containerID="df45616bb2f2b8038baaeda2c377e17ad320350a7daeedcc00faf4515dc2b4e7" exitCode=0
Jul 10 18:55:25.780343 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:25.779931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7" event={"ID":"e295dc0a-1c37-4fe6-bc4d-b41159a2d499","Type":"ContainerDied","Data":"df45616bb2f2b8038baaeda2c377e17ad320350a7daeedcc00faf4515dc2b4e7"}
Jul 10 18:55:26.786003 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:26.785965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7" event={"ID":"e295dc0a-1c37-4fe6-bc4d-b41159a2d499","Type":"ContainerStarted","Data":"a66445c5bd5c1a97320c2c9b4b691200c67d808ef312d25883358969454c5e8a"}
Jul 10 18:55:26.807280 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:26.807225    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7" podStartSLOduration=3.37419537 podStartE2EDuration="7.80720734s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.43301197s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:55:26.805270842 +0000 UTC m=+904.273189700" watchObservedRunningTime="2026-07-10 18:55:26.80720734 +0000 UTC m=+904.275126171"
Jul 10 18:55:30.279429 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:30.279391    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:30.279830 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:30.279452    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:30.292683 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:30.292652    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:30.814545 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:30.814514    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:32.724892 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:55:32.724862    2565 logging.go:55] [core] [Channel #30 SubChannel #31]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.28:9003", ServerName: "10.134.0.28:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.28:9003: connect: connection refused"
Jul 10 18:55:32.813292 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:32.813261    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5_bb846fb8-bd40-4c24-8d93-09cacea6d6e1/tokenizer/0.log"
Jul 10 18:55:32.813932 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:32.813909    2565 generic.go:356] "Generic (PLEG): container finished" podID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerID="d05ce77b681eb40e036ac48a86c7a1d4de5a44b8586a6ecd10555af0d1ba10a2" exitCode=137
Jul 10 18:55:32.814052 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:32.813978    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" event={"ID":"bb846fb8-bd40-4c24-8d93-09cacea6d6e1","Type":"ContainerDied","Data":"d05ce77b681eb40e036ac48a86c7a1d4de5a44b8586a6ecd10555af0d1ba10a2"}
Jul 10 18:55:33.380583 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.380559    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5_bb846fb8-bd40-4c24-8d93-09cacea6d6e1/tokenizer/0.log"
Jul 10 18:55:33.395999 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.395965    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:55:33.555621 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.555524    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dhlm\" (UniqueName: \"kubernetes.io/projected/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-kube-api-access-2dhlm\") pod \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") "
Jul 10 18:55:33.555621 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.555573    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-tmp\") pod \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") "
Jul 10 18:55:33.555621 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.555589    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-cache\") pod \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") "
Jul 10 18:55:33.555924 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.555633    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tls-certs\") pod \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") "
Jul 10 18:55:33.555924 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.555707    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-uds\") pod \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") "
Jul 10 18:55:33.555924 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.555775    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-kserve-provision-location\") pod \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\" (UID: \"bb846fb8-bd40-4c24-8d93-09cacea6d6e1\") "
Jul 10 18:55:33.555924 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.555905    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "bb846fb8-bd40-4c24-8d93-09cacea6d6e1" (UID: "bb846fb8-bd40-4c24-8d93-09cacea6d6e1"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:33.556222 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.555994    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "bb846fb8-bd40-4c24-8d93-09cacea6d6e1" (UID: "bb846fb8-bd40-4c24-8d93-09cacea6d6e1"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:33.556222 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.555997    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "bb846fb8-bd40-4c24-8d93-09cacea6d6e1" (UID: "bb846fb8-bd40-4c24-8d93-09cacea6d6e1"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:33.556222 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.556029    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:33.556541 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.556520    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb846fb8-bd40-4c24-8d93-09cacea6d6e1" (UID: "bb846fb8-bd40-4c24-8d93-09cacea6d6e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:33.557797 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.557776    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-kube-api-access-2dhlm" (OuterVolumeSpecName: "kube-api-access-2dhlm") pod "bb846fb8-bd40-4c24-8d93-09cacea6d6e1" (UID: "bb846fb8-bd40-4c24-8d93-09cacea6d6e1"). InnerVolumeSpecName "kube-api-access-2dhlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:55:33.557866 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.557803    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb846fb8-bd40-4c24-8d93-09cacea6d6e1" (UID: "bb846fb8-bd40-4c24-8d93-09cacea6d6e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:55:33.657431 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.657392    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:33.657431 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.657425    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-uds\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:33.657431 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.657435    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:33.657431 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.657444    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dhlm\" (UniqueName: \"kubernetes.io/projected/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-kube-api-access-2dhlm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:33.657702 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.657453    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb846fb8-bd40-4c24-8d93-09cacea6d6e1-tokenizer-tmp\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:33.725550 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.725504    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.28:9003\" within 1s: context deadline exceeded"
Jul 10 18:55:33.824247 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.824134    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5_bb846fb8-bd40-4c24-8d93-09cacea6d6e1/tokenizer/0.log"
Jul 10 18:55:33.824969 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.824948    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"
Jul 10 18:55:33.825101 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.824945    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5" event={"ID":"bb846fb8-bd40-4c24-8d93-09cacea6d6e1","Type":"ContainerDied","Data":"b5a347c0ba4efdc20fa64b83cf87569eb07dbe2ed8f55f83c25d687795989068"}
Jul 10 18:55:33.825158 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.825108    2565 scope.go:117] "RemoveContainer" containerID="d05ce77b681eb40e036ac48a86c7a1d4de5a44b8586a6ecd10555af0d1ba10a2"
Jul 10 18:55:33.834455 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.834431    2565 scope.go:117] "RemoveContainer" containerID="dfc0138753b936f722d951923dfc67475746d1663b1456bdb2f67b17673f4189"
Jul 10 18:55:33.841883 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.841868    2565 scope.go:117] "RemoveContainer" containerID="ee1f2732ad209343dc4b2b5c01982f4813a95104214abcc99ef5aff7b6e68590"
Jul 10 18:55:33.854552 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.854528    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"]
Jul 10 18:55:33.859486 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:33.859463    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-64c65bfb5bq5"]
Jul 10 18:55:35.094224 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:35.094189    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" path="/var/lib/kubelet/pods/bb846fb8-bd40-4c24-8d93-09cacea6d6e1/volumes"
Jul 10 18:55:53.376841 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.376809    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"]
Jul 10 18:55:53.377484 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.377031    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7" podUID="e295dc0a-1c37-4fe6-bc4d-b41159a2d499" containerName="main" containerID="cri-o://a66445c5bd5c1a97320c2c9b4b691200c67d808ef312d25883358969454c5e8a" gracePeriod=30
Jul 10 18:55:53.632592 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.632520    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:53.710232 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.710200    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-home\") pod \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") "
Jul 10 18:55:53.710232 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.710231    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-model-cache\") pod \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") "
Jul 10 18:55:53.710468 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.710260    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-dshm\") pod \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") "
Jul 10 18:55:53.710468 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.710283    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-tls-certs\") pod \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") "
Jul 10 18:55:53.710468 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.710309    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-tmp-dir\") pod \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") "
Jul 10 18:55:53.710468 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.710376    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gslgd\" (UniqueName: \"kubernetes.io/projected/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-kube-api-access-gslgd\") pod \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") "
Jul 10 18:55:53.710468 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.710409    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-kserve-provision-location\") pod \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\" (UID: \"e295dc0a-1c37-4fe6-bc4d-b41159a2d499\") "
Jul 10 18:55:53.710721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.710566    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-model-cache" (OuterVolumeSpecName: "model-cache") pod "e295dc0a-1c37-4fe6-bc4d-b41159a2d499" (UID: "e295dc0a-1c37-4fe6-bc4d-b41159a2d499"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:53.710721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.710559    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-home" (OuterVolumeSpecName: "home") pod "e295dc0a-1c37-4fe6-bc4d-b41159a2d499" (UID: "e295dc0a-1c37-4fe6-bc4d-b41159a2d499"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:53.710721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.710605    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e295dc0a-1c37-4fe6-bc4d-b41159a2d499" (UID: "e295dc0a-1c37-4fe6-bc4d-b41159a2d499"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:53.710721 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.710684    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:53.710876 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.710729    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:53.710876 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.710747    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:53.712542 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.712516    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-kube-api-access-gslgd" (OuterVolumeSpecName: "kube-api-access-gslgd") pod "e295dc0a-1c37-4fe6-bc4d-b41159a2d499" (UID: "e295dc0a-1c37-4fe6-bc4d-b41159a2d499"). InnerVolumeSpecName "kube-api-access-gslgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:55:53.712887 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.712866    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e295dc0a-1c37-4fe6-bc4d-b41159a2d499" (UID: "e295dc0a-1c37-4fe6-bc4d-b41159a2d499"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:55:53.712944 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.712883    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-dshm" (OuterVolumeSpecName: "dshm") pod "e295dc0a-1c37-4fe6-bc4d-b41159a2d499" (UID: "e295dc0a-1c37-4fe6-bc4d-b41159a2d499"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:53.765281 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.765241    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e295dc0a-1c37-4fe6-bc4d-b41159a2d499" (UID: "e295dc0a-1c37-4fe6-bc4d-b41159a2d499"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:53.811111 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.811079    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:53.811111 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.811105    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:53.811111 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.811115    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gslgd\" (UniqueName: \"kubernetes.io/projected/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-kube-api-access-gslgd\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:53.811363 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.811125    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e295dc0a-1c37-4fe6-bc4d-b41159a2d499-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:55:53.926202 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.926081    2565 generic.go:356] "Generic (PLEG): container finished" podID="e295dc0a-1c37-4fe6-bc4d-b41159a2d499" containerID="a66445c5bd5c1a97320c2c9b4b691200c67d808ef312d25883358969454c5e8a" exitCode=0
Jul 10 18:55:53.926202 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.926157    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"
Jul 10 18:55:53.926202 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.926177    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7" event={"ID":"e295dc0a-1c37-4fe6-bc4d-b41159a2d499","Type":"ContainerDied","Data":"a66445c5bd5c1a97320c2c9b4b691200c67d808ef312d25883358969454c5e8a"}
Jul 10 18:55:53.926472 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.926246    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7" event={"ID":"e295dc0a-1c37-4fe6-bc4d-b41159a2d499","Type":"ContainerDied","Data":"e6bc33565aa7cd41d63d40d3a0f116f8debb6eb661e0780d37800a0a6d4e2068"}
Jul 10 18:55:53.926472 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.926276    2565 scope.go:117] "RemoveContainer" containerID="a66445c5bd5c1a97320c2c9b4b691200c67d808ef312d25883358969454c5e8a"
Jul 10 18:55:53.935567 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.935551    2565 scope.go:117] "RemoveContainer" containerID="df45616bb2f2b8038baaeda2c377e17ad320350a7daeedcc00faf4515dc2b4e7"
Jul 10 18:55:53.949913 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.949888    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"]
Jul 10 18:55:53.950878 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.950862    2565 scope.go:117] "RemoveContainer" containerID="a66445c5bd5c1a97320c2c9b4b691200c67d808ef312d25883358969454c5e8a"
Jul 10 18:55:53.951208 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:55:53.951188    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66445c5bd5c1a97320c2c9b4b691200c67d808ef312d25883358969454c5e8a\": container with ID starting with a66445c5bd5c1a97320c2c9b4b691200c67d808ef312d25883358969454c5e8a not found: ID does not exist" containerID="a66445c5bd5c1a97320c2c9b4b691200c67d808ef312d25883358969454c5e8a"
Jul 10 18:55:53.951266 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.951219    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66445c5bd5c1a97320c2c9b4b691200c67d808ef312d25883358969454c5e8a"} err="failed to get container status \"a66445c5bd5c1a97320c2c9b4b691200c67d808ef312d25883358969454c5e8a\": rpc error: code = NotFound desc = could not find container \"a66445c5bd5c1a97320c2c9b4b691200c67d808ef312d25883358969454c5e8a\": container with ID starting with a66445c5bd5c1a97320c2c9b4b691200c67d808ef312d25883358969454c5e8a not found: ID does not exist"
Jul 10 18:55:53.951266 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.951238    2565 scope.go:117] "RemoveContainer" containerID="df45616bb2f2b8038baaeda2c377e17ad320350a7daeedcc00faf4515dc2b4e7"
Jul 10 18:55:53.951503 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:55:53.951485    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df45616bb2f2b8038baaeda2c377e17ad320350a7daeedcc00faf4515dc2b4e7\": container with ID starting with df45616bb2f2b8038baaeda2c377e17ad320350a7daeedcc00faf4515dc2b4e7 not found: ID does not exist" containerID="df45616bb2f2b8038baaeda2c377e17ad320350a7daeedcc00faf4515dc2b4e7"
Jul 10 18:55:53.951559 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.951512    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df45616bb2f2b8038baaeda2c377e17ad320350a7daeedcc00faf4515dc2b4e7"} err="failed to get container status \"df45616bb2f2b8038baaeda2c377e17ad320350a7daeedcc00faf4515dc2b4e7\": rpc error: code = NotFound desc = could not find container \"df45616bb2f2b8038baaeda2c377e17ad320350a7daeedcc00faf4515dc2b4e7\": container with ID starting with df45616bb2f2b8038baaeda2c377e17ad320350a7daeedcc00faf4515dc2b4e7 not found: ID does not exist"
Jul 10 18:55:53.953917 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:53.953896    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6657b58549-hk5t7"]
Jul 10 18:55:55.094351 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:55.094321    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e295dc0a-1c37-4fe6-bc4d-b41159a2d499" path="/var/lib/kubelet/pods/e295dc0a-1c37-4fe6-bc4d-b41159a2d499/volumes"
Jul 10 18:55:56.617781 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.617751    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"]
Jul 10 18:55:56.618264 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.618168    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="storage-initializer"
Jul 10 18:55:56.618264 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.618185    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="storage-initializer"
Jul 10 18:55:56.618264 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.618199    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e295dc0a-1c37-4fe6-bc4d-b41159a2d499" containerName="storage-initializer"
Jul 10 18:55:56.618264 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.618206    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e295dc0a-1c37-4fe6-bc4d-b41159a2d499" containerName="storage-initializer"
Jul 10 18:55:56.618264 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.618219    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="main"
Jul 10 18:55:56.618264 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.618226    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="main"
Jul 10 18:55:56.618264 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.618234    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e295dc0a-1c37-4fe6-bc4d-b41159a2d499" containerName="main"
Jul 10 18:55:56.618264 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.618241    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e295dc0a-1c37-4fe6-bc4d-b41159a2d499" containerName="main"
Jul 10 18:55:56.618264 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.618259    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="tokenizer"
Jul 10 18:55:56.618264 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.618267    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="tokenizer"
Jul 10 18:55:56.618712 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.618343    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="tokenizer"
Jul 10 18:55:56.618712 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.618356    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb846fb8-bd40-4c24-8d93-09cacea6d6e1" containerName="main"
Jul 10 18:55:56.618712 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.618365    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e295dc0a-1c37-4fe6-bc4d-b41159a2d499" containerName="main"
Jul 10 18:55:56.622328 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.622308    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.625194 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.625174    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 18:55:56.625297 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.625246    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 10 18:55:56.625410 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.625397    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 18:55:56.626433 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.626418    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jjdpb\""
Jul 10 18:55:56.629462 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.629442    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-kserve-provision-location\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.629547 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.629482    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-model-cache\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.629547 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.629526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-tmp-dir\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.629663 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.629555    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nwdd\" (UniqueName: \"kubernetes.io/projected/1faef3fd-5254-48ee-8cd7-0816d6610310-kube-api-access-7nwdd\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.629663 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.629584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-dshm\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.629663 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.629630    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-home\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.629792 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.629662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1faef3fd-5254-48ee-8cd7-0816d6610310-tls-certs\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.634511 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.634491    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"]
Jul 10 18:55:56.730590 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.730558    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-dshm\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.730590 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.730593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-home\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.730825 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.730618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1faef3fd-5254-48ee-8cd7-0816d6610310-tls-certs\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.730825 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.730655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-kserve-provision-location\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.730825 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.730679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-model-cache\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.730825 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.730699    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-tmp-dir\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.730825 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.730774    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nwdd\" (UniqueName: \"kubernetes.io/projected/1faef3fd-5254-48ee-8cd7-0816d6610310-kube-api-access-7nwdd\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.731049 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.731025    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-home\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.731106 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.731026    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-kserve-provision-location\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.731194 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.731121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-model-cache\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.731307 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.731265    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-tmp-dir\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.732886 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.732859    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-dshm\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.733129 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.733115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1faef3fd-5254-48ee-8cd7-0816d6610310-tls-certs\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.738901 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.738880    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nwdd\" (UniqueName: \"kubernetes.io/projected/1faef3fd-5254-48ee-8cd7-0816d6610310-kube-api-access-7nwdd\") pod \"stop-feature-test-kserve-f9cdc8f48-tn5vx\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:56.933437 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:56.933353    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:55:57.078000 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:57.077974    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"]
Jul 10 18:55:57.080602 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:55:57.080570    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faef3fd_5254_48ee_8cd7_0816d6610310.slice/crio-30a6a4f77baa36dfd248a4740628ff3e53a5588f09b21fad3fa25be44c795102 WatchSource:0}: Error finding container 30a6a4f77baa36dfd248a4740628ff3e53a5588f09b21fad3fa25be44c795102: Status 404 returned error can't find the container with id 30a6a4f77baa36dfd248a4740628ff3e53a5588f09b21fad3fa25be44c795102
Jul 10 18:55:57.952089 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:57.952048    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" event={"ID":"1faef3fd-5254-48ee-8cd7-0816d6610310","Type":"ContainerStarted","Data":"8b55a9c11eccf924904331244e87c2edc119c996edf23df03dd49b2839187e72"}
Jul 10 18:55:57.952616 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:55:57.952111    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" event={"ID":"1faef3fd-5254-48ee-8cd7-0816d6610310","Type":"ContainerStarted","Data":"30a6a4f77baa36dfd248a4740628ff3e53a5588f09b21fad3fa25be44c795102"}
Jul 10 18:56:01.972685 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:56:01.972649    2565 generic.go:356] "Generic (PLEG): container finished" podID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerID="8b55a9c11eccf924904331244e87c2edc119c996edf23df03dd49b2839187e72" exitCode=0
Jul 10 18:56:01.973060 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:56:01.972710    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" event={"ID":"1faef3fd-5254-48ee-8cd7-0816d6610310","Type":"ContainerDied","Data":"8b55a9c11eccf924904331244e87c2edc119c996edf23df03dd49b2839187e72"}
Jul 10 18:56:30.128068 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:56:30.128023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" event={"ID":"1faef3fd-5254-48ee-8cd7-0816d6610310","Type":"ContainerStarted","Data":"9df97913afd5fc2acc2746c6da2d0330176960787f9f4fd254192028e3493b38"}
Jul 10 18:56:30.153672 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:56:30.153608    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" podStartSLOduration=2.558209439 podStartE2EDuration="34.1535844s" totalImagesPullingTime="27.222722225s" totalInitContainerRuntime="4.372652736s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:56:30.150881495 +0000 UTC m=+967.618800330" watchObservedRunningTime="2026-07-10 18:56:30.1535844 +0000 UTC m=+967.621503235"
Jul 10 18:56:36.934220 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:56:36.934179    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:56:36.934714 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:56:36.934281    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:56:36.935779 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:56:36.935746    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 10 18:56:46.934262 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:56:46.934203    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 10 18:56:56.933850 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:56:56.933797    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 10 18:57:06.934642 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:57:06.934600    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 10 18:57:16.934016 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:57:16.933969    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 10 18:57:26.934803 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:57:26.934754    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 10 18:57:36.934246 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:57:36.934203    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 10 18:57:46.934655 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:57:46.934607    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 10 18:57:56.934210 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:57:56.934130    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 10 18:58:04.450112 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.450080    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"]
Jul 10 18:58:04.454142 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.454122    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.458050 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.458030    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
Jul 10 18:58:04.470674 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.470644    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"]
Jul 10 18:58:04.521137 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.521093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpzq5\" (UniqueName: \"kubernetes.io/projected/0ce7dba9-570e-4f4e-87e5-73af4457683f-kube-api-access-bpzq5\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.521137 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.521134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.521438 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.521184    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.521438 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.521231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.521438 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.521270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ce7dba9-570e-4f4e-87e5-73af4457683f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.521438 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.521344    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.521438 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.521379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.622646 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.622588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpzq5\" (UniqueName: \"kubernetes.io/projected/0ce7dba9-570e-4f4e-87e5-73af4457683f-kube-api-access-bpzq5\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.622859 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.622666    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.622859 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.622696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.622859 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.622732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.622859 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.622807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ce7dba9-570e-4f4e-87e5-73af4457683f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.623059 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.622906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.623059 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.622942    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.623197 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.623173    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.623260 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.623245    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.623369 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.623348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.623411 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.623350    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.625163 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.625128    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.625610 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.625589    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ce7dba9-570e-4f4e-87e5-73af4457683f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.631957 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.631927    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpzq5\" (UniqueName: \"kubernetes.io/projected/0ce7dba9-570e-4f4e-87e5-73af4457683f-kube-api-access-bpzq5\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.764624 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.764592    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:04.768035 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.767630    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"]
Jul 10 18:58:04.771400 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.771380    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.774389 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.774366    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa-dockercfg-sxfqx\""
Jul 10 18:58:04.787514 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.787482    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"]
Jul 10 18:58:04.824116 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.824086    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.824272 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.824221    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cp9j\" (UniqueName: \"kubernetes.io/projected/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-kube-api-access-5cp9j\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.824343 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.824313    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.824408 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.824396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.824465 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.824426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.824563 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.824519    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.912432 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.912312    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"]
Jul 10 18:58:04.915356 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:58:04.915319    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce7dba9_570e_4f4e_87e5_73af4457683f.slice/crio-315cd2f6a8a163ce37723b8b1b662b941388fc455135d0588b3a1cd38cafbcf2 WatchSource:0}: Error finding container 315cd2f6a8a163ce37723b8b1b662b941388fc455135d0588b3a1cd38cafbcf2: Status 404 returned error can't find the container with id 315cd2f6a8a163ce37723b8b1b662b941388fc455135d0588b3a1cd38cafbcf2
Jul 10 18:58:04.924967 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.924945    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.925032 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.924977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.925032 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.925020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.925100 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.925064    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.925134 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.925105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cp9j\" (UniqueName: \"kubernetes.io/projected/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-kube-api-access-5cp9j\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.925196 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.925162    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.925513 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.925482    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.925513 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.925498    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.925646 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.925522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.925646 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.925576    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.927476 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.927450    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:04.934997 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:04.934968    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cp9j\" (UniqueName: \"kubernetes.io/projected/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-kube-api-access-5cp9j\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:05.098618 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:05.098510    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:05.292415 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:05.292387    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"]
Jul 10 18:58:05.294616 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:58:05.294587    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f2df5c_f613_4ba2_b070_b9a24446bcd4.slice/crio-e7c8577731e4e19eafa9f44c7f0fca0a3b59552fec54f144250d7996a1922110 WatchSource:0}: Error finding container e7c8577731e4e19eafa9f44c7f0fca0a3b59552fec54f144250d7996a1922110: Status 404 returned error can't find the container with id e7c8577731e4e19eafa9f44c7f0fca0a3b59552fec54f144250d7996a1922110
Jul 10 18:58:05.574228 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:05.574185    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp" event={"ID":"a3f2df5c-f613-4ba2-b070-b9a24446bcd4","Type":"ContainerStarted","Data":"46bb30e06793fcdfa97b2718d47c0e706b73ab1bcaa49bb5fe5d0f038d0e2565"}
Jul 10 18:58:05.574801 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:05.574770    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp" event={"ID":"a3f2df5c-f613-4ba2-b070-b9a24446bcd4","Type":"ContainerStarted","Data":"e7c8577731e4e19eafa9f44c7f0fca0a3b59552fec54f144250d7996a1922110"}
Jul 10 18:58:05.575961 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:05.575938    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" event={"ID":"0ce7dba9-570e-4f4e-87e5-73af4457683f","Type":"ContainerStarted","Data":"dd10641f814211f6b4b5788a1b9fefc6d96b332d525281219099eed7bf2634c9"}
Jul 10 18:58:05.576075 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:05.575986    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" event={"ID":"0ce7dba9-570e-4f4e-87e5-73af4457683f","Type":"ContainerStarted","Data":"315cd2f6a8a163ce37723b8b1b662b941388fc455135d0588b3a1cd38cafbcf2"}
Jul 10 18:58:06.583478 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:06.583439    2565 generic.go:356] "Generic (PLEG): container finished" podID="a3f2df5c-f613-4ba2-b070-b9a24446bcd4" containerID="46bb30e06793fcdfa97b2718d47c0e706b73ab1bcaa49bb5fe5d0f038d0e2565" exitCode=0
Jul 10 18:58:06.583929 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:06.583527    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp" event={"ID":"a3f2df5c-f613-4ba2-b070-b9a24446bcd4","Type":"ContainerDied","Data":"46bb30e06793fcdfa97b2718d47c0e706b73ab1bcaa49bb5fe5d0f038d0e2565"}
Jul 10 18:58:06.944488 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:06.944415    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:58:06.952606 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:06.952580    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:58:07.589562 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:07.589525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp" event={"ID":"a3f2df5c-f613-4ba2-b070-b9a24446bcd4","Type":"ContainerStarted","Data":"7a235662c3c5a03a0ac0132e10bd375752a4f8da214219321291afba9065930a"}
Jul 10 18:58:07.590063 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:07.589591    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp" event={"ID":"a3f2df5c-f613-4ba2-b070-b9a24446bcd4","Type":"ContainerStarted","Data":"7acb4cf8dd999475c2f950a94e8b626182d4f931c105a6c10c83d2b2c112d049"}
Jul 10 18:58:07.590063 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:07.589752    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:07.620376 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:07.620324    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp" podStartSLOduration=2.89678696 podStartE2EDuration="3.620307312s" totalImagesPullingTime="0s" totalInitContainerRuntime="723.520352ms" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:07.618177813 +0000 UTC m=+1065.086096671" watchObservedRunningTime="2026-07-10 18:58:07.620307312 +0000 UTC m=+1065.088226144"
Jul 10 18:58:10.606285 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:10.606248    2565 generic.go:356] "Generic (PLEG): container finished" podID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerID="dd10641f814211f6b4b5788a1b9fefc6d96b332d525281219099eed7bf2634c9" exitCode=0
Jul 10 18:58:10.606829 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:10.606297    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" event={"ID":"0ce7dba9-570e-4f4e-87e5-73af4457683f","Type":"ContainerDied","Data":"dd10641f814211f6b4b5788a1b9fefc6d96b332d525281219099eed7bf2634c9"}
Jul 10 18:58:11.613124 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:11.613089    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" event={"ID":"0ce7dba9-570e-4f4e-87e5-73af4457683f","Type":"ContainerStarted","Data":"4a4e459a0bdda316c250f5c62ec354a1a5e80b3dfe029d91f4e2fbbd64315e4b"}
Jul 10 18:58:11.639086 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:11.639032    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" podStartSLOduration=2.601045435 podStartE2EDuration="7.639014892s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.037969457s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:11.637007095 +0000 UTC m=+1069.104925955" watchObservedRunningTime="2026-07-10 18:58:11.639014892 +0000 UTC m=+1069.106933725"
Jul 10 18:58:14.765233 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:14.765131    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:14.765233 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:14.765202    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 18:58:14.766974 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:14.766943    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 10 18:58:15.099434 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:15.099347    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:15.099434 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:15.099423    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:15.102390 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:15.102364    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:15.639217 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:15.639177    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:15.684098 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:15.684054    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"]
Jul 10 18:58:15.684399 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:15.684369    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="main" containerID="cri-o://9df97913afd5fc2acc2746c6da2d0330176960787f9f4fd254192028e3493b38" gracePeriod=30
Jul 10 18:58:24.765141 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:24.765083    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 10 18:58:34.765458 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:34.765407    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 10 18:58:36.324264 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.324222    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"]
Jul 10 18:58:36.327799 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.327771    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.354750 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.354716    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"]
Jul 10 18:58:36.400022 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.399980    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-tmp-dir\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.400022 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.400022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-home\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.400277 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.400045    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-dshm\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.400277 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.400122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-model-cache\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.400277 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.400178    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-kserve-provision-location\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.400277 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.400214    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b884e8c-0822-4778-85d0-37ce9cdf3fef-tls-certs\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.400277 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.400264    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5cwb\" (UniqueName: \"kubernetes.io/projected/1b884e8c-0822-4778-85d0-37ce9cdf3fef-kube-api-access-p5cwb\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.501193 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.501132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5cwb\" (UniqueName: \"kubernetes.io/projected/1b884e8c-0822-4778-85d0-37ce9cdf3fef-kube-api-access-p5cwb\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.501386 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.501296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-tmp-dir\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.501386 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.501326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-home\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.501386 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.501343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-dshm\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.501386 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.501384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-model-cache\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.501583 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.501417    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-kserve-provision-location\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.501583 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.501453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b884e8c-0822-4778-85d0-37ce9cdf3fef-tls-certs\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.508662 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.502181    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-tmp-dir\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.508662 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.502268    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-home\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.508662 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.502292    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-model-cache\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.508662 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.502440    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-kserve-provision-location\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.509396 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.509371    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-dshm\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.511257 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.511234    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b884e8c-0822-4778-85d0-37ce9cdf3fef-tls-certs\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.512381 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.512349    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5cwb\" (UniqueName: \"kubernetes.io/projected/1b884e8c-0822-4778-85d0-37ce9cdf3fef-kube-api-access-p5cwb\") pod \"stop-feature-test-kserve-f9cdc8f48-92dw2\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.637874 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.637789    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:36.638907 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.638884    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 18:58:36.813522 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:36.813493    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"]
Jul 10 18:58:36.816429 ip-10-0-132-91 kubenswrapper[2565]: W0710 18:58:36.816394    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b884e8c_0822_4778_85d0_37ce9cdf3fef.slice/crio-b827a6292aae749a1561e15643b4e9bec765add12e77b4edd676613ba43f85c4 WatchSource:0}: Error finding container b827a6292aae749a1561e15643b4e9bec765add12e77b4edd676613ba43f85c4: Status 404 returned error can't find the container with id b827a6292aae749a1561e15643b4e9bec765add12e77b4edd676613ba43f85c4
Jul 10 18:58:37.740421 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:37.740381    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" event={"ID":"1b884e8c-0822-4778-85d0-37ce9cdf3fef","Type":"ContainerStarted","Data":"e041c5379de6ed50d4c6aeac28400464898c0184db5e0f529cf897526a38c1f9"}
Jul 10 18:58:37.740843 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:37.740435    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" event={"ID":"1b884e8c-0822-4778-85d0-37ce9cdf3fef","Type":"ContainerStarted","Data":"b827a6292aae749a1561e15643b4e9bec765add12e77b4edd676613ba43f85c4"}
Jul 10 18:58:41.763611 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:41.763572    2565 generic.go:356] "Generic (PLEG): container finished" podID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerID="e041c5379de6ed50d4c6aeac28400464898c0184db5e0f529cf897526a38c1f9" exitCode=0
Jul 10 18:58:41.764035 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:41.763648    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" event={"ID":"1b884e8c-0822-4778-85d0-37ce9cdf3fef","Type":"ContainerDied","Data":"e041c5379de6ed50d4c6aeac28400464898c0184db5e0f529cf897526a38c1f9"}
Jul 10 18:58:42.771078 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:42.771036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" event={"ID":"1b884e8c-0822-4778-85d0-37ce9cdf3fef","Type":"ContainerStarted","Data":"df6e244e4386969876850f231b63d7284347efbcbe5b641918baf34878221285"}
Jul 10 18:58:42.795879 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:42.795817    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" podStartSLOduration=2.284752458 podStartE2EDuration="6.795798628s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.51104617s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:42.792913793 +0000 UTC m=+1100.260832660" watchObservedRunningTime="2026-07-10 18:58:42.795798628 +0000 UTC m=+1100.263717460"
Jul 10 18:58:44.766006 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:44.765954    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 10 18:58:45.941978 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:45.941910    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-f9cdc8f48-tn5vx_1faef3fd-5254-48ee-8cd7-0816d6610310/main/0.log"
Jul 10 18:58:45.943718 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:45.943696    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:58:45.985242 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:45.985173    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-dshm\") pod \"1faef3fd-5254-48ee-8cd7-0816d6610310\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") "
Jul 10 18:58:45.985242 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:45.985218    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1faef3fd-5254-48ee-8cd7-0816d6610310-tls-certs\") pod \"1faef3fd-5254-48ee-8cd7-0816d6610310\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") "
Jul 10 18:58:45.985510 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:45.985290    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-model-cache\") pod \"1faef3fd-5254-48ee-8cd7-0816d6610310\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") "
Jul 10 18:58:45.985510 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:45.985321    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-tmp-dir\") pod \"1faef3fd-5254-48ee-8cd7-0816d6610310\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") "
Jul 10 18:58:45.985510 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:45.985345    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nwdd\" (UniqueName: \"kubernetes.io/projected/1faef3fd-5254-48ee-8cd7-0816d6610310-kube-api-access-7nwdd\") pod \"1faef3fd-5254-48ee-8cd7-0816d6610310\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") "
Jul 10 18:58:45.985510 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:45.985373    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-home\") pod \"1faef3fd-5254-48ee-8cd7-0816d6610310\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") "
Jul 10 18:58:45.985510 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:45.985412    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-kserve-provision-location\") pod \"1faef3fd-5254-48ee-8cd7-0816d6610310\" (UID: \"1faef3fd-5254-48ee-8cd7-0816d6610310\") "
Jul 10 18:58:45.985767 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:45.985667    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-model-cache" (OuterVolumeSpecName: "model-cache") pod "1faef3fd-5254-48ee-8cd7-0816d6610310" (UID: "1faef3fd-5254-48ee-8cd7-0816d6610310"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:58:45.986587 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:45.986537    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-home" (OuterVolumeSpecName: "home") pod "1faef3fd-5254-48ee-8cd7-0816d6610310" (UID: "1faef3fd-5254-48ee-8cd7-0816d6610310"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:58:46.007088 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:45.991293    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-dshm" (OuterVolumeSpecName: "dshm") pod "1faef3fd-5254-48ee-8cd7-0816d6610310" (UID: "1faef3fd-5254-48ee-8cd7-0816d6610310"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:58:46.007088 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:45.991524    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1faef3fd-5254-48ee-8cd7-0816d6610310-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1faef3fd-5254-48ee-8cd7-0816d6610310" (UID: "1faef3fd-5254-48ee-8cd7-0816d6610310"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:58:46.007088 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:45.999174    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1faef3fd-5254-48ee-8cd7-0816d6610310" (UID: "1faef3fd-5254-48ee-8cd7-0816d6610310"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:58:46.007088 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.007015    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faef3fd-5254-48ee-8cd7-0816d6610310-kube-api-access-7nwdd" (OuterVolumeSpecName: "kube-api-access-7nwdd") pod "1faef3fd-5254-48ee-8cd7-0816d6610310" (UID: "1faef3fd-5254-48ee-8cd7-0816d6610310"). InnerVolumeSpecName "kube-api-access-7nwdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:58:46.063945 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.063893    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1faef3fd-5254-48ee-8cd7-0816d6610310" (UID: "1faef3fd-5254-48ee-8cd7-0816d6610310"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:58:46.086274 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.086229    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:58:46.086274 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.086271    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1faef3fd-5254-48ee-8cd7-0816d6610310-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:58:46.086475 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.086288    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:58:46.086475 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.086302    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:58:46.086475 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.086318    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nwdd\" (UniqueName: \"kubernetes.io/projected/1faef3fd-5254-48ee-8cd7-0816d6610310-kube-api-access-7nwdd\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:58:46.086475 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.086334    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:58:46.086475 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.086348    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faef3fd-5254-48ee-8cd7-0816d6610310-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 18:58:46.638160 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.638105    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:46.638363 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.638202    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 18:58:46.639688 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.639652    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 10 18:58:46.797212 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.797175    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-f9cdc8f48-tn5vx_1faef3fd-5254-48ee-8cd7-0816d6610310/main/0.log"
Jul 10 18:58:46.797613 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.797589    2565 generic.go:356] "Generic (PLEG): container finished" podID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerID="9df97913afd5fc2acc2746c6da2d0330176960787f9f4fd254192028e3493b38" exitCode=137
Jul 10 18:58:46.797734 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.797647    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" event={"ID":"1faef3fd-5254-48ee-8cd7-0816d6610310","Type":"ContainerDied","Data":"9df97913afd5fc2acc2746c6da2d0330176960787f9f4fd254192028e3493b38"}
Jul 10 18:58:46.797734 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.797662    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"
Jul 10 18:58:46.797734 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.797703    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx" event={"ID":"1faef3fd-5254-48ee-8cd7-0816d6610310","Type":"ContainerDied","Data":"30a6a4f77baa36dfd248a4740628ff3e53a5588f09b21fad3fa25be44c795102"}
Jul 10 18:58:46.797919 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.797735    2565 scope.go:117] "RemoveContainer" containerID="9df97913afd5fc2acc2746c6da2d0330176960787f9f4fd254192028e3493b38"
Jul 10 18:58:46.806524 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.806500    2565 scope.go:117] "RemoveContainer" containerID="8b55a9c11eccf924904331244e87c2edc119c996edf23df03dd49b2839187e72"
Jul 10 18:58:46.817246 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.817219    2565 scope.go:117] "RemoveContainer" containerID="9df97913afd5fc2acc2746c6da2d0330176960787f9f4fd254192028e3493b38"
Jul 10 18:58:46.817562 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:58:46.817539    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df97913afd5fc2acc2746c6da2d0330176960787f9f4fd254192028e3493b38\": container with ID starting with 9df97913afd5fc2acc2746c6da2d0330176960787f9f4fd254192028e3493b38 not found: ID does not exist" containerID="9df97913afd5fc2acc2746c6da2d0330176960787f9f4fd254192028e3493b38"
Jul 10 18:58:46.817636 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.817574    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df97913afd5fc2acc2746c6da2d0330176960787f9f4fd254192028e3493b38"} err="failed to get container status \"9df97913afd5fc2acc2746c6da2d0330176960787f9f4fd254192028e3493b38\": rpc error: code = NotFound desc = could not find container \"9df97913afd5fc2acc2746c6da2d0330176960787f9f4fd254192028e3493b38\": container with ID starting with 9df97913afd5fc2acc2746c6da2d0330176960787f9f4fd254192028e3493b38 not found: ID does not exist"
Jul 10 18:58:46.817636 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.817596    2565 scope.go:117] "RemoveContainer" containerID="8b55a9c11eccf924904331244e87c2edc119c996edf23df03dd49b2839187e72"
Jul 10 18:58:46.817842 ip-10-0-132-91 kubenswrapper[2565]: E0710 18:58:46.817818    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b55a9c11eccf924904331244e87c2edc119c996edf23df03dd49b2839187e72\": container with ID starting with 8b55a9c11eccf924904331244e87c2edc119c996edf23df03dd49b2839187e72 not found: ID does not exist" containerID="8b55a9c11eccf924904331244e87c2edc119c996edf23df03dd49b2839187e72"
Jul 10 18:58:46.817918 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.817851    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b55a9c11eccf924904331244e87c2edc119c996edf23df03dd49b2839187e72"} err="failed to get container status \"8b55a9c11eccf924904331244e87c2edc119c996edf23df03dd49b2839187e72\": rpc error: code = NotFound desc = could not find container \"8b55a9c11eccf924904331244e87c2edc119c996edf23df03dd49b2839187e72\": container with ID starting with 8b55a9c11eccf924904331244e87c2edc119c996edf23df03dd49b2839187e72 not found: ID does not exist"
Jul 10 18:58:46.825754 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.825729    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"]
Jul 10 18:58:46.830038 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:46.830011    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-tn5vx"]
Jul 10 18:58:47.095141 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:47.095108    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" path="/var/lib/kubelet/pods/1faef3fd-5254-48ee-8cd7-0816d6610310/volumes"
Jul 10 18:58:54.766041 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:54.765978    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 10 18:58:56.638275 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:58:56.638225    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 10 18:59:04.765592 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:59:04.765539    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 10 18:59:06.638127 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:59:06.638085    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 10 18:59:14.765531 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:59:14.765485    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 10 18:59:16.638280 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:59:16.638234    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 10 18:59:24.765988 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:59:24.765936    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 10 18:59:26.638352 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:59:26.638306    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 10 18:59:34.765834 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:59:34.765775    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 10 18:59:36.638242 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:59:36.638200    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 10 18:59:44.765848 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:59:44.765747    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 10 18:59:46.638084 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:59:46.638044    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 10 18:59:54.765403 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:59:54.765343    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 10 18:59:56.638236 ip-10-0-132-91 kubenswrapper[2565]: I0710 18:59:56.638201    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 10 19:00:04.775701 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:04.775671    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 19:00:04.783952 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:04.783923    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 19:00:06.638207 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:06.638164    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 10 19:00:13.912654 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:13.912609    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"]
Jul 10 19:00:13.913027 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:13.912864    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp" podUID="a3f2df5c-f613-4ba2-b070-b9a24446bcd4" containerName="main" containerID="cri-o://7acb4cf8dd999475c2f950a94e8b626182d4f931c105a6c10c83d2b2c112d049" gracePeriod=30
Jul 10 19:00:13.913069 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:13.912988    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp" podUID="a3f2df5c-f613-4ba2-b070-b9a24446bcd4" containerName="tokenizer" containerID="cri-o://7a235662c3c5a03a0ac0132e10bd375752a4f8da214219321291afba9065930a" gracePeriod=30
Jul 10 19:00:13.931497 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:13.930865    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"]
Jul 10 19:00:13.931497 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:13.931106    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main" containerID="cri-o://4a4e459a0bdda316c250f5c62ec354a1a5e80b3dfe029d91f4e2fbbd64315e4b" gracePeriod=30
Jul 10 19:00:14.227469 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:14.227372    2565 generic.go:356] "Generic (PLEG): container finished" podID="a3f2df5c-f613-4ba2-b070-b9a24446bcd4" containerID="7acb4cf8dd999475c2f950a94e8b626182d4f931c105a6c10c83d2b2c112d049" exitCode=0
Jul 10 19:00:14.227469 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:14.227441    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp" event={"ID":"a3f2df5c-f613-4ba2-b070-b9a24446bcd4","Type":"ContainerDied","Data":"7acb4cf8dd999475c2f950a94e8b626182d4f931c105a6c10c83d2b2c112d049"}
Jul 10 19:00:15.186220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.186195    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 19:00:15.234596 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.234565    2565 generic.go:356] "Generic (PLEG): container finished" podID="a3f2df5c-f613-4ba2-b070-b9a24446bcd4" containerID="7a235662c3c5a03a0ac0132e10bd375752a4f8da214219321291afba9065930a" exitCode=0
Jul 10 19:00:15.234793 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.234633    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"
Jul 10 19:00:15.234793 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.234654    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp" event={"ID":"a3f2df5c-f613-4ba2-b070-b9a24446bcd4","Type":"ContainerDied","Data":"7a235662c3c5a03a0ac0132e10bd375752a4f8da214219321291afba9065930a"}
Jul 10 19:00:15.234793 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.234727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp" event={"ID":"a3f2df5c-f613-4ba2-b070-b9a24446bcd4","Type":"ContainerDied","Data":"e7c8577731e4e19eafa9f44c7f0fca0a3b59552fec54f144250d7996a1922110"}
Jul 10 19:00:15.234793 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.234752    2565 scope.go:117] "RemoveContainer" containerID="7a235662c3c5a03a0ac0132e10bd375752a4f8da214219321291afba9065930a"
Jul 10 19:00:15.243890 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.243873    2565 scope.go:117] "RemoveContainer" containerID="7acb4cf8dd999475c2f950a94e8b626182d4f931c105a6c10c83d2b2c112d049"
Jul 10 19:00:15.251406 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.251386    2565 scope.go:117] "RemoveContainer" containerID="46bb30e06793fcdfa97b2718d47c0e706b73ab1bcaa49bb5fe5d0f038d0e2565"
Jul 10 19:00:15.258625 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.258609    2565 scope.go:117] "RemoveContainer" containerID="7a235662c3c5a03a0ac0132e10bd375752a4f8da214219321291afba9065930a"
Jul 10 19:00:15.258860 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:00:15.258845    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a235662c3c5a03a0ac0132e10bd375752a4f8da214219321291afba9065930a\": container with ID starting with 7a235662c3c5a03a0ac0132e10bd375752a4f8da214219321291afba9065930a not found: ID does not exist" containerID="7a235662c3c5a03a0ac0132e10bd375752a4f8da214219321291afba9065930a"
Jul 10 19:00:15.258909 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.258868    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a235662c3c5a03a0ac0132e10bd375752a4f8da214219321291afba9065930a"} err="failed to get container status \"7a235662c3c5a03a0ac0132e10bd375752a4f8da214219321291afba9065930a\": rpc error: code = NotFound desc = could not find container \"7a235662c3c5a03a0ac0132e10bd375752a4f8da214219321291afba9065930a\": container with ID starting with 7a235662c3c5a03a0ac0132e10bd375752a4f8da214219321291afba9065930a not found: ID does not exist"
Jul 10 19:00:15.258909 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.258885    2565 scope.go:117] "RemoveContainer" containerID="7acb4cf8dd999475c2f950a94e8b626182d4f931c105a6c10c83d2b2c112d049"
Jul 10 19:00:15.259118 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:00:15.259098    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7acb4cf8dd999475c2f950a94e8b626182d4f931c105a6c10c83d2b2c112d049\": container with ID starting with 7acb4cf8dd999475c2f950a94e8b626182d4f931c105a6c10c83d2b2c112d049 not found: ID does not exist" containerID="7acb4cf8dd999475c2f950a94e8b626182d4f931c105a6c10c83d2b2c112d049"
Jul 10 19:00:15.259197 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.259125    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7acb4cf8dd999475c2f950a94e8b626182d4f931c105a6c10c83d2b2c112d049"} err="failed to get container status \"7acb4cf8dd999475c2f950a94e8b626182d4f931c105a6c10c83d2b2c112d049\": rpc error: code = NotFound desc = could not find container \"7acb4cf8dd999475c2f950a94e8b626182d4f931c105a6c10c83d2b2c112d049\": container with ID starting with 7acb4cf8dd999475c2f950a94e8b626182d4f931c105a6c10c83d2b2c112d049 not found: ID does not exist"
Jul 10 19:00:15.259197 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.259140    2565 scope.go:117] "RemoveContainer" containerID="46bb30e06793fcdfa97b2718d47c0e706b73ab1bcaa49bb5fe5d0f038d0e2565"
Jul 10 19:00:15.259377 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:00:15.259360    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46bb30e06793fcdfa97b2718d47c0e706b73ab1bcaa49bb5fe5d0f038d0e2565\": container with ID starting with 46bb30e06793fcdfa97b2718d47c0e706b73ab1bcaa49bb5fe5d0f038d0e2565 not found: ID does not exist" containerID="46bb30e06793fcdfa97b2718d47c0e706b73ab1bcaa49bb5fe5d0f038d0e2565"
Jul 10 19:00:15.259420 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.259382    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bb30e06793fcdfa97b2718d47c0e706b73ab1bcaa49bb5fe5d0f038d0e2565"} err="failed to get container status \"46bb30e06793fcdfa97b2718d47c0e706b73ab1bcaa49bb5fe5d0f038d0e2565\": rpc error: code = NotFound desc = could not find container \"46bb30e06793fcdfa97b2718d47c0e706b73ab1bcaa49bb5fe5d0f038d0e2565\": container with ID starting with 46bb30e06793fcdfa97b2718d47c0e706b73ab1bcaa49bb5fe5d0f038d0e2565 not found: ID does not exist"
Jul 10 19:00:15.366283 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.366174    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-uds\") pod \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") "
Jul 10 19:00:15.366283 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.366249    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tls-certs\") pod \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") "
Jul 10 19:00:15.366515 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.366288    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cp9j\" (UniqueName: \"kubernetes.io/projected/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-kube-api-access-5cp9j\") pod \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") "
Jul 10 19:00:15.366515 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.366316    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-tmp\") pod \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") "
Jul 10 19:00:15.366515 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.366358    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-kserve-provision-location\") pod \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") "
Jul 10 19:00:15.366515 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.366404    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-cache\") pod \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\" (UID: \"a3f2df5c-f613-4ba2-b070-b9a24446bcd4\") "
Jul 10 19:00:15.366515 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.366493    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a3f2df5c-f613-4ba2-b070-b9a24446bcd4" (UID: "a3f2df5c-f613-4ba2-b070-b9a24446bcd4"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:15.366749 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.366648    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-uds\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:15.366749 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.366722    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "a3f2df5c-f613-4ba2-b070-b9a24446bcd4" (UID: "a3f2df5c-f613-4ba2-b070-b9a24446bcd4"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:15.367087 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.367059    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "a3f2df5c-f613-4ba2-b070-b9a24446bcd4" (UID: "a3f2df5c-f613-4ba2-b070-b9a24446bcd4"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:15.367577 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.367553    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a3f2df5c-f613-4ba2-b070-b9a24446bcd4" (UID: "a3f2df5c-f613-4ba2-b070-b9a24446bcd4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:15.368814 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.368791    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-kube-api-access-5cp9j" (OuterVolumeSpecName: "kube-api-access-5cp9j") pod "a3f2df5c-f613-4ba2-b070-b9a24446bcd4" (UID: "a3f2df5c-f613-4ba2-b070-b9a24446bcd4"). InnerVolumeSpecName "kube-api-access-5cp9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:00:15.368981 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.368966    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3f2df5c-f613-4ba2-b070-b9a24446bcd4" (UID: "a3f2df5c-f613-4ba2-b070-b9a24446bcd4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:15.467698 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.467657    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:15.467698 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.467683    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cp9j\" (UniqueName: \"kubernetes.io/projected/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-kube-api-access-5cp9j\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:15.467698 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.467692    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-tmp\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:15.467698 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.467703    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:15.467698 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.467712    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3f2df5c-f613-4ba2-b070-b9a24446bcd4-tokenizer-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:15.562154 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.562118    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"]
Jul 10 19:00:15.564712 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:15.564687    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schewh7bp"]
Jul 10 19:00:16.638802 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:16.638756    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 10 19:00:17.094516 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.094481    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3f2df5c-f613-4ba2-b070-b9a24446bcd4" path="/var/lib/kubelet/pods/a3f2df5c-f613-4ba2-b070-b9a24446bcd4/volumes"
Jul 10 19:00:17.247640 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.247603    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"]
Jul 10 19:00:17.248041 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.248027    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="storage-initializer"
Jul 10 19:00:17.248094 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.248045    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="storage-initializer"
Jul 10 19:00:17.248094 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.248062    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="main"
Jul 10 19:00:17.248094 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.248071    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="main"
Jul 10 19:00:17.248094 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.248081    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f2df5c-f613-4ba2-b070-b9a24446bcd4" containerName="storage-initializer"
Jul 10 19:00:17.248094 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.248090    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f2df5c-f613-4ba2-b070-b9a24446bcd4" containerName="storage-initializer"
Jul 10 19:00:17.248289 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.248117    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f2df5c-f613-4ba2-b070-b9a24446bcd4" containerName="tokenizer"
Jul 10 19:00:17.248289 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.248125    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f2df5c-f613-4ba2-b070-b9a24446bcd4" containerName="tokenizer"
Jul 10 19:00:17.248289 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.248140    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f2df5c-f613-4ba2-b070-b9a24446bcd4" containerName="main"
Jul 10 19:00:17.248289 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.248164    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f2df5c-f613-4ba2-b070-b9a24446bcd4" containerName="main"
Jul 10 19:00:17.248289 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.248232    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f2df5c-f613-4ba2-b070-b9a24446bcd4" containerName="tokenizer"
Jul 10 19:00:17.248289 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.248243    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1faef3fd-5254-48ee-8cd7-0816d6610310" containerName="main"
Jul 10 19:00:17.248289 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.248252    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f2df5c-f613-4ba2-b070-b9a24446bcd4" containerName="main"
Jul 10 19:00:17.254503 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.254480    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.257553 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.257537    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 10 19:00:17.281854 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.281828    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"]
Jul 10 19:00:17.283285 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.283259    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ab6bb30-96de-4b6b-b300-aaf51440df63-tls-certs\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.283406 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.283304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-dshm\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.283406 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.283330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrsmt\" (UniqueName: \"kubernetes.io/projected/3ab6bb30-96de-4b6b-b300-aaf51440df63-kube-api-access-lrsmt\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.283406 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.283392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-home\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.283562 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.283421    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-tmp-dir\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.283562 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.283439    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.283562 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.283455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-model-cache\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.384057 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.383968    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-home\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.384057 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.384023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-tmp-dir\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.384057 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.384050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.384366 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.384075    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-model-cache\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.384366 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.384108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ab6bb30-96de-4b6b-b300-aaf51440df63-tls-certs\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.384366 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.384233    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-dshm\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.384366 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.384258    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrsmt\" (UniqueName: \"kubernetes.io/projected/3ab6bb30-96de-4b6b-b300-aaf51440df63-kube-api-access-lrsmt\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.384587 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.384475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-home\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.384587 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.384526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.384587 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.384568    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-model-cache\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.384692 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.384667    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-tmp-dir\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.386677 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.386648    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-dshm\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.386883 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.386867    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ab6bb30-96de-4b6b-b300-aaf51440df63-tls-certs\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.393449 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.393429    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrsmt\" (UniqueName: \"kubernetes.io/projected/3ab6bb30-96de-4b6b-b300-aaf51440df63-kube-api-access-lrsmt\") pod \"custom-route-timeout-test-kserve-fb9546c6d-zzrhg\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.564836 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.564801    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:17.715460 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:17.715436    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"]
Jul 10 19:00:17.718702 ip-10-0-132-91 kubenswrapper[2565]: W0710 19:00:17.718669    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab6bb30_96de_4b6b_b300_aaf51440df63.slice/crio-0207b7f1d7a0b85a93234cedb3e91eb609c791c944c34153876101182fa2818d WatchSource:0}: Error finding container 0207b7f1d7a0b85a93234cedb3e91eb609c791c944c34153876101182fa2818d: Status 404 returned error can't find the container with id 0207b7f1d7a0b85a93234cedb3e91eb609c791c944c34153876101182fa2818d
Jul 10 19:00:18.249983 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:18.249946    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" event={"ID":"3ab6bb30-96de-4b6b-b300-aaf51440df63","Type":"ContainerStarted","Data":"0f95271acd16cf7bcd2c42c53d644e0997ea69fb852708113d421c971b2df4cb"}
Jul 10 19:00:18.250218 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:18.250005    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" event={"ID":"3ab6bb30-96de-4b6b-b300-aaf51440df63","Type":"ContainerStarted","Data":"0207b7f1d7a0b85a93234cedb3e91eb609c791c944c34153876101182fa2818d"}
Jul 10 19:00:22.282484 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:22.282443    2565 generic.go:356] "Generic (PLEG): container finished" podID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerID="0f95271acd16cf7bcd2c42c53d644e0997ea69fb852708113d421c971b2df4cb" exitCode=0
Jul 10 19:00:22.282958 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:22.282492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" event={"ID":"3ab6bb30-96de-4b6b-b300-aaf51440df63","Type":"ContainerDied","Data":"0f95271acd16cf7bcd2c42c53d644e0997ea69fb852708113d421c971b2df4cb"}
Jul 10 19:00:23.288910 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:23.288856    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" event={"ID":"3ab6bb30-96de-4b6b-b300-aaf51440df63","Type":"ContainerStarted","Data":"a02755a92a6b961c032664388d1bad6964c20dc74b057efa6cf2ba66e5aa8e17"}
Jul 10 19:00:23.315073 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:23.315010    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" podStartSLOduration=1.9673389970000001 podStartE2EDuration="6.314993434s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.347654437s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:23.312590177 +0000 UTC m=+1200.780509044" watchObservedRunningTime="2026-07-10 19:00:23.314993434 +0000 UTC m=+1200.782912320"
Jul 10 19:00:26.639270 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:26.639210    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Jul 10 19:00:27.565350 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:27.565307    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:27.565545 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:27.565390    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:00:27.566385 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:27.566357    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 10 19:00:36.648731 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:36.648703    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 19:00:36.656980 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:36.656952    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 19:00:37.565302 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:37.565262    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 10 19:00:37.912662 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:37.912570    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"]
Jul 10 19:00:38.361330 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:38.361264    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main" containerID="cri-o://df6e244e4386969876850f231b63d7284347efbcbe5b641918baf34878221285" gracePeriod=30
Jul 10 19:00:44.201405 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.201380    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq_0ce7dba9-570e-4f4e-87e5-73af4457683f/main/0.log"
Jul 10 19:00:44.203193 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.203178    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 19:00:44.330987 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.330951    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-model-cache\") pod \"0ce7dba9-570e-4f4e-87e5-73af4457683f\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") "
Jul 10 19:00:44.331209 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.331013    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpzq5\" (UniqueName: \"kubernetes.io/projected/0ce7dba9-570e-4f4e-87e5-73af4457683f-kube-api-access-bpzq5\") pod \"0ce7dba9-570e-4f4e-87e5-73af4457683f\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") "
Jul 10 19:00:44.331209 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.331036    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-dshm\") pod \"0ce7dba9-570e-4f4e-87e5-73af4457683f\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") "
Jul 10 19:00:44.331209 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.331062    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-kserve-provision-location\") pod \"0ce7dba9-570e-4f4e-87e5-73af4457683f\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") "
Jul 10 19:00:44.331209 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.331089    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-home\") pod \"0ce7dba9-570e-4f4e-87e5-73af4457683f\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") "
Jul 10 19:00:44.331209 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.331128    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ce7dba9-570e-4f4e-87e5-73af4457683f-tls-certs\") pod \"0ce7dba9-570e-4f4e-87e5-73af4457683f\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") "
Jul 10 19:00:44.331209 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.331185    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-tmp-dir\") pod \"0ce7dba9-570e-4f4e-87e5-73af4457683f\" (UID: \"0ce7dba9-570e-4f4e-87e5-73af4457683f\") "
Jul 10 19:00:44.331502 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.331238    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ce7dba9-570e-4f4e-87e5-73af4457683f" (UID: "0ce7dba9-570e-4f4e-87e5-73af4457683f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:44.331502 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.331471    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:44.331896 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.331864    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-home" (OuterVolumeSpecName: "home") pod "0ce7dba9-570e-4f4e-87e5-73af4457683f" (UID: "0ce7dba9-570e-4f4e-87e5-73af4457683f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:44.333411 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.333376    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ce7dba9-570e-4f4e-87e5-73af4457683f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ce7dba9-570e-4f4e-87e5-73af4457683f" (UID: "0ce7dba9-570e-4f4e-87e5-73af4457683f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:44.333595 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.333570    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-dshm" (OuterVolumeSpecName: "dshm") pod "0ce7dba9-570e-4f4e-87e5-73af4457683f" (UID: "0ce7dba9-570e-4f4e-87e5-73af4457683f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:44.333807 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.333781    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce7dba9-570e-4f4e-87e5-73af4457683f-kube-api-access-bpzq5" (OuterVolumeSpecName: "kube-api-access-bpzq5") pod "0ce7dba9-570e-4f4e-87e5-73af4457683f" (UID: "0ce7dba9-570e-4f4e-87e5-73af4457683f"). InnerVolumeSpecName "kube-api-access-bpzq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:00:44.346755 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.346726    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ce7dba9-570e-4f4e-87e5-73af4457683f" (UID: "0ce7dba9-570e-4f4e-87e5-73af4457683f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:44.387159 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.387099    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0ce7dba9-570e-4f4e-87e5-73af4457683f" (UID: "0ce7dba9-570e-4f4e-87e5-73af4457683f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:44.396495 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.396462    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq_0ce7dba9-570e-4f4e-87e5-73af4457683f/main/0.log"
Jul 10 19:00:44.396846 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.396826    2565 generic.go:356] "Generic (PLEG): container finished" podID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerID="4a4e459a0bdda316c250f5c62ec354a1a5e80b3dfe029d91f4e2fbbd64315e4b" exitCode=137
Jul 10 19:00:44.396932 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.396893    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"
Jul 10 19:00:44.396932 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.396907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" event={"ID":"0ce7dba9-570e-4f4e-87e5-73af4457683f","Type":"ContainerDied","Data":"4a4e459a0bdda316c250f5c62ec354a1a5e80b3dfe029d91f4e2fbbd64315e4b"}
Jul 10 19:00:44.396997 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.396970    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq" event={"ID":"0ce7dba9-570e-4f4e-87e5-73af4457683f","Type":"ContainerDied","Data":"315cd2f6a8a163ce37723b8b1b662b941388fc455135d0588b3a1cd38cafbcf2"}
Jul 10 19:00:44.397030 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.397004    2565 scope.go:117] "RemoveContainer" containerID="4a4e459a0bdda316c250f5c62ec354a1a5e80b3dfe029d91f4e2fbbd64315e4b"
Jul 10 19:00:44.406211 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.406178    2565 scope.go:117] "RemoveContainer" containerID="dd10641f814211f6b4b5788a1b9fefc6d96b332d525281219099eed7bf2634c9"
Jul 10 19:00:44.415657 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.415637    2565 scope.go:117] "RemoveContainer" containerID="4a4e459a0bdda316c250f5c62ec354a1a5e80b3dfe029d91f4e2fbbd64315e4b"
Jul 10 19:00:44.415886 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:00:44.415867    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4e459a0bdda316c250f5c62ec354a1a5e80b3dfe029d91f4e2fbbd64315e4b\": container with ID starting with 4a4e459a0bdda316c250f5c62ec354a1a5e80b3dfe029d91f4e2fbbd64315e4b not found: ID does not exist" containerID="4a4e459a0bdda316c250f5c62ec354a1a5e80b3dfe029d91f4e2fbbd64315e4b"
Jul 10 19:00:44.415972 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.415893    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4e459a0bdda316c250f5c62ec354a1a5e80b3dfe029d91f4e2fbbd64315e4b"} err="failed to get container status \"4a4e459a0bdda316c250f5c62ec354a1a5e80b3dfe029d91f4e2fbbd64315e4b\": rpc error: code = NotFound desc = could not find container \"4a4e459a0bdda316c250f5c62ec354a1a5e80b3dfe029d91f4e2fbbd64315e4b\": container with ID starting with 4a4e459a0bdda316c250f5c62ec354a1a5e80b3dfe029d91f4e2fbbd64315e4b not found: ID does not exist"
Jul 10 19:00:44.415972 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.415912    2565 scope.go:117] "RemoveContainer" containerID="dd10641f814211f6b4b5788a1b9fefc6d96b332d525281219099eed7bf2634c9"
Jul 10 19:00:44.416136 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:00:44.416119    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd10641f814211f6b4b5788a1b9fefc6d96b332d525281219099eed7bf2634c9\": container with ID starting with dd10641f814211f6b4b5788a1b9fefc6d96b332d525281219099eed7bf2634c9 not found: ID does not exist" containerID="dd10641f814211f6b4b5788a1b9fefc6d96b332d525281219099eed7bf2634c9"
Jul 10 19:00:44.416235 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.416140    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd10641f814211f6b4b5788a1b9fefc6d96b332d525281219099eed7bf2634c9"} err="failed to get container status \"dd10641f814211f6b4b5788a1b9fefc6d96b332d525281219099eed7bf2634c9\": rpc error: code = NotFound desc = could not find container \"dd10641f814211f6b4b5788a1b9fefc6d96b332d525281219099eed7bf2634c9\": container with ID starting with dd10641f814211f6b4b5788a1b9fefc6d96b332d525281219099eed7bf2634c9 not found: ID does not exist"
Jul 10 19:00:44.425064 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.425037    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"]
Jul 10 19:00:44.429774 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.429749    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57dcfb8796mcxpq"]
Jul 10 19:00:44.432399 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.432375    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpzq5\" (UniqueName: \"kubernetes.io/projected/0ce7dba9-570e-4f4e-87e5-73af4457683f-kube-api-access-bpzq5\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:44.432399 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.432396    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:44.432564 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.432408    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:44.432564 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.432416    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:44.432564 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.432425    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ce7dba9-570e-4f4e-87e5-73af4457683f-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:44.432564 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:44.432433    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ce7dba9-570e-4f4e-87e5-73af4457683f-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:00:45.094680 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:45.094646    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" path="/var/lib/kubelet/pods/0ce7dba9-570e-4f4e-87e5-73af4457683f/volumes"
Jul 10 19:00:47.565653 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:47.565609    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 10 19:00:57.565492 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:00:57.565428    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 10 19:01:07.566283 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:07.566229    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 10 19:01:08.643943 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.643915    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-f9cdc8f48-92dw2_1b884e8c-0822-4778-85d0-37ce9cdf3fef/main/0.log"
Jul 10 19:01:08.645608 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.645592    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 19:01:08.724751 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.724710    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-tmp-dir\") pod \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") "
Jul 10 19:01:08.724930 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.724775    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5cwb\" (UniqueName: \"kubernetes.io/projected/1b884e8c-0822-4778-85d0-37ce9cdf3fef-kube-api-access-p5cwb\") pod \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") "
Jul 10 19:01:08.724930 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.724840    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-model-cache\") pod \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") "
Jul 10 19:01:08.724930 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.724869    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-home\") pod \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") "
Jul 10 19:01:08.724930 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.724911    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-kserve-provision-location\") pod \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") "
Jul 10 19:01:08.725192 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.724957    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-dshm\") pod \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") "
Jul 10 19:01:08.725192 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.724982    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b884e8c-0822-4778-85d0-37ce9cdf3fef-tls-certs\") pod \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\" (UID: \"1b884e8c-0822-4778-85d0-37ce9cdf3fef\") "
Jul 10 19:01:08.725301 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.725197    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b884e8c-0822-4778-85d0-37ce9cdf3fef" (UID: "1b884e8c-0822-4778-85d0-37ce9cdf3fef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:01:08.725601 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.725571    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-home" (OuterVolumeSpecName: "home") pod "1b884e8c-0822-4778-85d0-37ce9cdf3fef" (UID: "1b884e8c-0822-4778-85d0-37ce9cdf3fef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:01:08.727516 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.727482    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b884e8c-0822-4778-85d0-37ce9cdf3fef-kube-api-access-p5cwb" (OuterVolumeSpecName: "kube-api-access-p5cwb") pod "1b884e8c-0822-4778-85d0-37ce9cdf3fef" (UID: "1b884e8c-0822-4778-85d0-37ce9cdf3fef"). InnerVolumeSpecName "kube-api-access-p5cwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:01:08.727625 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.727560    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b884e8c-0822-4778-85d0-37ce9cdf3fef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b884e8c-0822-4778-85d0-37ce9cdf3fef" (UID: "1b884e8c-0822-4778-85d0-37ce9cdf3fef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:01:08.728275 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.728247    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-dshm" (OuterVolumeSpecName: "dshm") pod "1b884e8c-0822-4778-85d0-37ce9cdf3fef" (UID: "1b884e8c-0822-4778-85d0-37ce9cdf3fef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:01:08.738261 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.738224    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b884e8c-0822-4778-85d0-37ce9cdf3fef" (UID: "1b884e8c-0822-4778-85d0-37ce9cdf3fef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:01:08.787484 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.787437    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b884e8c-0822-4778-85d0-37ce9cdf3fef" (UID: "1b884e8c-0822-4778-85d0-37ce9cdf3fef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:01:08.826010 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.825975    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:01:08.826010 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.826007    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:01:08.826010 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.826017    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:01:08.826236 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.826025    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b884e8c-0822-4778-85d0-37ce9cdf3fef-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:01:08.826236 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.826033    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:01:08.826236 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.826042    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5cwb\" (UniqueName: \"kubernetes.io/projected/1b884e8c-0822-4778-85d0-37ce9cdf3fef-kube-api-access-p5cwb\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:01:08.826236 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:08.826049    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b884e8c-0822-4778-85d0-37ce9cdf3fef-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:01:09.513322 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:09.513296    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-f9cdc8f48-92dw2_1b884e8c-0822-4778-85d0-37ce9cdf3fef/main/0.log"
Jul 10 19:01:09.513735 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:09.513714    2565 generic.go:356] "Generic (PLEG): container finished" podID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerID="df6e244e4386969876850f231b63d7284347efbcbe5b641918baf34878221285" exitCode=137
Jul 10 19:01:09.513821 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:09.513785    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"
Jul 10 19:01:09.513821 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:09.513805    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" event={"ID":"1b884e8c-0822-4778-85d0-37ce9cdf3fef","Type":"ContainerDied","Data":"df6e244e4386969876850f231b63d7284347efbcbe5b641918baf34878221285"}
Jul 10 19:01:09.513904 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:09.513892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2" event={"ID":"1b884e8c-0822-4778-85d0-37ce9cdf3fef","Type":"ContainerDied","Data":"b827a6292aae749a1561e15643b4e9bec765add12e77b4edd676613ba43f85c4"}
Jul 10 19:01:09.513938 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:09.513920    2565 scope.go:117] "RemoveContainer" containerID="df6e244e4386969876850f231b63d7284347efbcbe5b641918baf34878221285"
Jul 10 19:01:09.522633 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:09.522611    2565 scope.go:117] "RemoveContainer" containerID="e041c5379de6ed50d4c6aeac28400464898c0184db5e0f529cf897526a38c1f9"
Jul 10 19:01:09.536446 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:09.536421    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"]
Jul 10 19:01:09.542037 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:09.542018    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f9cdc8f48-92dw2"]
Jul 10 19:01:09.589443 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:09.589423    2565 scope.go:117] "RemoveContainer" containerID="df6e244e4386969876850f231b63d7284347efbcbe5b641918baf34878221285"
Jul 10 19:01:09.589786 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:01:09.589760    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df6e244e4386969876850f231b63d7284347efbcbe5b641918baf34878221285\": container with ID starting with df6e244e4386969876850f231b63d7284347efbcbe5b641918baf34878221285 not found: ID does not exist" containerID="df6e244e4386969876850f231b63d7284347efbcbe5b641918baf34878221285"
Jul 10 19:01:09.589904 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:09.589790    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df6e244e4386969876850f231b63d7284347efbcbe5b641918baf34878221285"} err="failed to get container status \"df6e244e4386969876850f231b63d7284347efbcbe5b641918baf34878221285\": rpc error: code = NotFound desc = could not find container \"df6e244e4386969876850f231b63d7284347efbcbe5b641918baf34878221285\": container with ID starting with df6e244e4386969876850f231b63d7284347efbcbe5b641918baf34878221285 not found: ID does not exist"
Jul 10 19:01:09.589904 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:09.589809    2565 scope.go:117] "RemoveContainer" containerID="e041c5379de6ed50d4c6aeac28400464898c0184db5e0f529cf897526a38c1f9"
Jul 10 19:01:09.590134 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:01:09.590115    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e041c5379de6ed50d4c6aeac28400464898c0184db5e0f529cf897526a38c1f9\": container with ID starting with e041c5379de6ed50d4c6aeac28400464898c0184db5e0f529cf897526a38c1f9 not found: ID does not exist" containerID="e041c5379de6ed50d4c6aeac28400464898c0184db5e0f529cf897526a38c1f9"
Jul 10 19:01:09.590218 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:09.590142    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e041c5379de6ed50d4c6aeac28400464898c0184db5e0f529cf897526a38c1f9"} err="failed to get container status \"e041c5379de6ed50d4c6aeac28400464898c0184db5e0f529cf897526a38c1f9\": rpc error: code = NotFound desc = could not find container \"e041c5379de6ed50d4c6aeac28400464898c0184db5e0f529cf897526a38c1f9\": container with ID starting with e041c5379de6ed50d4c6aeac28400464898c0184db5e0f529cf897526a38c1f9 not found: ID does not exist"
Jul 10 19:01:11.094450 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:11.094415    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" path="/var/lib/kubelet/pods/1b884e8c-0822-4778-85d0-37ce9cdf3fef/volumes"
Jul 10 19:01:17.566298 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:17.566212    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 10 19:01:27.565774 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:27.565727    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 10 19:01:37.566105 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:37.566066    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 10 19:01:47.565792 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:47.565741    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 10 19:01:57.575739 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:57.575708    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:01:57.583467 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:01:57.583448    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:02:21.939312 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.939267    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"]
Jul 10 19:02:21.939764 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.939734    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="storage-initializer"
Jul 10 19:02:21.939764 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.939752    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="storage-initializer"
Jul 10 19:02:21.939839 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.939771    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main"
Jul 10 19:02:21.939839 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.939782    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main"
Jul 10 19:02:21.939839 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.939796    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main"
Jul 10 19:02:21.939839 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.939805    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main"
Jul 10 19:02:21.939839 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.939827    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="storage-initializer"
Jul 10 19:02:21.939839 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.939836    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="storage-initializer"
Jul 10 19:02:21.940010 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.939914    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b884e8c-0822-4778-85d0-37ce9cdf3fef" containerName="main"
Jul 10 19:02:21.940010 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.939929    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce7dba9-570e-4f4e-87e5-73af4457683f" containerName="main"
Jul 10 19:02:21.942345 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.942324    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:21.945461 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.945435    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-t9vdn\""
Jul 10 19:02:21.945593 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.945517    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 10 19:02:21.960375 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.960344    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"]
Jul 10 19:02:21.967931 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.967909    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"]
Jul 10 19:02:21.970234 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.970213    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:21.989540 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:21.989510    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"]
Jul 10 19:02:22.012337 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.012311    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-home\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.012484 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.012345    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-model-cache\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.012484 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.012363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pm27\" (UniqueName: \"kubernetes.io/projected/b4c36103-2327-4049-877e-47fe04e8b9c2-kube-api-access-8pm27\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.012484 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.012420    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.012484 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.012454    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-dshm\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.012484 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.012479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.012684 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.012498    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-home\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.012684 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.012515    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.012684 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.012558    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.012684 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.012577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.012684 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.012604    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfkf9\" (UniqueName: \"kubernetes.io/projected/f161594b-469d-4330-bb0e-ba97bd29874c-kube-api-access-hfkf9\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.012684 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.012630    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c36103-2327-4049-877e-47fe04e8b9c2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.012684 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.012666    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.012890 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.012702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f161594b-469d-4330-bb0e-ba97bd29874c-tls-certs\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.113912 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.113878    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f161594b-469d-4330-bb0e-ba97bd29874c-tls-certs\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.114089 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.113922    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-home\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.114089 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.113951    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-model-cache\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.114089 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.113975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pm27\" (UniqueName: \"kubernetes.io/projected/b4c36103-2327-4049-877e-47fe04e8b9c2-kube-api-access-8pm27\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.114089 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.114285 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114252    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-dshm\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.114327 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.114369 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-home\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.114369 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-home\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.114369 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114360    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.114521 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.114521 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114405    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.114521 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114403    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-model-cache\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.114521 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.114722 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfkf9\" (UniqueName: \"kubernetes.io/projected/f161594b-469d-4330-bb0e-ba97bd29874c-kube-api-access-hfkf9\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.114722 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c36103-2327-4049-877e-47fe04e8b9c2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.114722 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114594    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-home\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.114722 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.114722 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.114722 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114712    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.115037 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114737    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.115037 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.114955    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.116532 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.116510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f161594b-469d-4330-bb0e-ba97bd29874c-tls-certs\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.116654 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.116510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-dshm\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.116896 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.116880    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.116939 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.116896    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c36103-2327-4049-877e-47fe04e8b9c2-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.126440 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.126415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pm27\" (UniqueName: \"kubernetes.io/projected/b4c36103-2327-4049-877e-47fe04e8b9c2-kube-api-access-8pm27\") pod \"router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.126541 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.126524    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfkf9\" (UniqueName: \"kubernetes.io/projected/f161594b-469d-4330-bb0e-ba97bd29874c-kube-api-access-hfkf9\") pod \"router-with-refs-pd-test-kserve-6fd47695bb-j7qrz\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.253137 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.253102    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:22.282321 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.282285    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:22.423367 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.421171    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"]
Jul 10 19:02:22.424297 ip-10-0-132-91 kubenswrapper[2565]: W0710 19:02:22.424270    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf161594b_469d_4330_bb0e_ba97bd29874c.slice/crio-6206b98dac24765208e2b3e711195bdba807d9ee918183921baf25041c13c107 WatchSource:0}: Error finding container 6206b98dac24765208e2b3e711195bdba807d9ee918183921baf25041c13c107: Status 404 returned error can't find the container with id 6206b98dac24765208e2b3e711195bdba807d9ee918183921baf25041c13c107
Jul 10 19:02:22.425924 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.425905    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:02:22.447080 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.447049    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"]
Jul 10 19:02:22.450914 ip-10-0-132-91 kubenswrapper[2565]: W0710 19:02:22.450883    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c36103_2327_4049_877e_47fe04e8b9c2.slice/crio-fb11d59eebcafc290fad1101ba99b0af1e2ef8be4d006018069cdf39086fa74a WatchSource:0}: Error finding container fb11d59eebcafc290fad1101ba99b0af1e2ef8be4d006018069cdf39086fa74a: Status 404 returned error can't find the container with id fb11d59eebcafc290fad1101ba99b0af1e2ef8be4d006018069cdf39086fa74a
Jul 10 19:02:22.836812 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.836578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" event={"ID":"b4c36103-2327-4049-877e-47fe04e8b9c2","Type":"ContainerStarted","Data":"15bec37aebdf93d79be559b29dcd139f2922a7642b36ff971131bbe08cfa6811"}
Jul 10 19:02:22.836812 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.836649    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" event={"ID":"b4c36103-2327-4049-877e-47fe04e8b9c2","Type":"ContainerStarted","Data":"fb11d59eebcafc290fad1101ba99b0af1e2ef8be4d006018069cdf39086fa74a"}
Jul 10 19:02:22.838182 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:22.838114    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" event={"ID":"f161594b-469d-4330-bb0e-ba97bd29874c","Type":"ContainerStarted","Data":"6206b98dac24765208e2b3e711195bdba807d9ee918183921baf25041c13c107"}
Jul 10 19:02:23.843927 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:23.843883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" event={"ID":"f161594b-469d-4330-bb0e-ba97bd29874c","Type":"ContainerStarted","Data":"40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f"}
Jul 10 19:02:24.851794 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:24.851761    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" event={"ID":"f161594b-469d-4330-bb0e-ba97bd29874c","Type":"ContainerStarted","Data":"4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028"}
Jul 10 19:02:24.852331 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:24.852017    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:27.870534 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:27.870491    2565 generic.go:356] "Generic (PLEG): container finished" podID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerID="15bec37aebdf93d79be559b29dcd139f2922a7642b36ff971131bbe08cfa6811" exitCode=0
Jul 10 19:02:27.870534 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:27.870524    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" event={"ID":"b4c36103-2327-4049-877e-47fe04e8b9c2","Type":"ContainerDied","Data":"15bec37aebdf93d79be559b29dcd139f2922a7642b36ff971131bbe08cfa6811"}
Jul 10 19:02:28.881216 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:28.881134    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" event={"ID":"b4c36103-2327-4049-877e-47fe04e8b9c2","Type":"ContainerStarted","Data":"670922b34ae81f875cb96addb5a40b9baa37d87794d6847860fdbc2d0842d3ac"}
Jul 10 19:02:28.884257 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:28.884220    2565 generic.go:356] "Generic (PLEG): container finished" podID="f161594b-469d-4330-bb0e-ba97bd29874c" containerID="4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028" exitCode=0
Jul 10 19:02:28.884420 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:28.884269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" event={"ID":"f161594b-469d-4330-bb0e-ba97bd29874c","Type":"ContainerDied","Data":"4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028"}
Jul 10 19:02:28.914219 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:28.914118    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podStartSLOduration=3.477451894 podStartE2EDuration="7.914089389s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.436637495s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:02:28.910578413 +0000 UTC m=+1326.378497275" watchObservedRunningTime="2026-07-10 19:02:28.914089389 +0000 UTC m=+1326.382008237"
Jul 10 19:02:29.892826 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:29.892795    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" event={"ID":"f161594b-469d-4330-bb0e-ba97bd29874c","Type":"ContainerStarted","Data":"9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5"}
Jul 10 19:02:29.921937 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:29.921885    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podStartSLOduration=3.608386134 podStartE2EDuration="8.921863458s" totalImagesPullingTime="790.283044ms" totalInitContainerRuntime="4.52319428s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:02:29.920400911 +0000 UTC m=+1327.388319766" watchObservedRunningTime="2026-07-10 19:02:29.921863458 +0000 UTC m=+1327.389782291"
Jul 10 19:02:32.253503 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:32.253463    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:32.253908 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:32.253535    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:32.254978 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:32.254944    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:02:32.282800 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:32.282773    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:32.282932 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:32.282835    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:02:32.284196 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:32.284165    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:02:42.254561 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:42.254515    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:02:42.267128 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:42.267095    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:02:42.282987 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:42.282946    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:02:52.253673 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:52.253560    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:02:52.283376 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:02:52.283327    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:03:02.254287 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:02.254241    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:03:02.283403 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:02.283358    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:03:02.856977 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:02.856937    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"]
Jul 10 19:03:02.857266 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:02.857232    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="main" containerID="cri-o://a02755a92a6b961c032664388d1bad6964c20dc74b057efa6cf2ba66e5aa8e17" gracePeriod=30
Jul 10 19:03:10.714598 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.714559    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"]
Jul 10 19:03:10.717093 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.717072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.719927 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.719905    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 10 19:03:10.732178 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.732141    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"]
Jul 10 19:03:10.765487 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.765461    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46b0492-ba9a-4580-bccc-da507c3f55ab-tls-certs\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.765487 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.765490    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-home\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.765727 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.765508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-tmp-dir\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.765727 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.765552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-model-cache\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.765727 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.765592    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6mbj\" (UniqueName: \"kubernetes.io/projected/f46b0492-ba9a-4580-bccc-da507c3f55ab-kube-api-access-d6mbj\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.765727 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.765616    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-kserve-provision-location\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.765727 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.765637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-dshm\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.866886 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.866844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46b0492-ba9a-4580-bccc-da507c3f55ab-tls-certs\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.866886 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.866876    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-home\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.866886 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.866893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-tmp-dir\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.867231 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.866924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-model-cache\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.867231 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.867043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6mbj\" (UniqueName: \"kubernetes.io/projected/f46b0492-ba9a-4580-bccc-da507c3f55ab-kube-api-access-d6mbj\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.867231 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.867103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-kserve-provision-location\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.867231 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.867139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-dshm\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.867471 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.867345    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-tmp-dir\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.867742 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.867694    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-model-cache\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.867742 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.867733    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-home\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.867948 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.867923    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-kserve-provision-location\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.869773 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.869745    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-dshm\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.870185 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.870138    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46b0492-ba9a-4580-bccc-da507c3f55ab-tls-certs\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:10.878106 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:10.878063    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6mbj\" (UniqueName: \"kubernetes.io/projected/f46b0492-ba9a-4580-bccc-da507c3f55ab-kube-api-access-d6mbj\") pod \"router-with-refs-test-kserve-76b768cb56-46p5r\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:11.028117 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:11.028089    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:11.184517 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:11.184480    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"]
Jul 10 19:03:11.188448 ip-10-0-132-91 kubenswrapper[2565]: W0710 19:03:11.188407    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46b0492_ba9a_4580_bccc_da507c3f55ab.slice/crio-17d648f93a5c522e480ae0591a258cb58bd7b409a223f2c37f8bc92ff0b403ce WatchSource:0}: Error finding container 17d648f93a5c522e480ae0591a258cb58bd7b409a223f2c37f8bc92ff0b403ce: Status 404 returned error can't find the container with id 17d648f93a5c522e480ae0591a258cb58bd7b409a223f2c37f8bc92ff0b403ce
Jul 10 19:03:12.110227 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:12.110183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" event={"ID":"f46b0492-ba9a-4580-bccc-da507c3f55ab","Type":"ContainerStarted","Data":"c1fc6ee1975257b48452c61510c035023e0aa6f2c6d08fdb89b602a1fc915832"}
Jul 10 19:03:12.110721 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:12.110247    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" event={"ID":"f46b0492-ba9a-4580-bccc-da507c3f55ab","Type":"ContainerStarted","Data":"17d648f93a5c522e480ae0591a258cb58bd7b409a223f2c37f8bc92ff0b403ce"}
Jul 10 19:03:12.254662 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:12.254238    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:03:12.283320 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:12.283273    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:03:16.135440 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:16.135399    2565 generic.go:356] "Generic (PLEG): container finished" podID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerID="c1fc6ee1975257b48452c61510c035023e0aa6f2c6d08fdb89b602a1fc915832" exitCode=0
Jul 10 19:03:16.135872 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:16.135459    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" event={"ID":"f46b0492-ba9a-4580-bccc-da507c3f55ab","Type":"ContainerDied","Data":"c1fc6ee1975257b48452c61510c035023e0aa6f2c6d08fdb89b602a1fc915832"}
Jul 10 19:03:17.143222 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:17.143177    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" event={"ID":"f46b0492-ba9a-4580-bccc-da507c3f55ab","Type":"ContainerStarted","Data":"9f0fb9c5ccce5aae0d40476255cefb2a1acdb4523e6a8449a688ff11e393768a"}
Jul 10 19:03:21.029819 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:21.028945    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:21.029819 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:21.029764    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:03:21.030364 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:21.029876    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 10 19:03:22.254093 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:22.254048    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:03:22.282852 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:22.282815    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:03:31.029100 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:31.029045    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 10 19:03:32.253544 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:32.253502    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:03:32.283754 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:32.283708    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:03:33.216979 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.216948    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-fb9546c6d-zzrhg_3ab6bb30-96de-4b6b-b300-aaf51440df63/main/0.log"
Jul 10 19:03:33.218585 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.218563    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:03:33.227976 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.227955    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-fb9546c6d-zzrhg_3ab6bb30-96de-4b6b-b300-aaf51440df63/main/0.log"
Jul 10 19:03:33.228353 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.228328    2565 generic.go:356] "Generic (PLEG): container finished" podID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerID="a02755a92a6b961c032664388d1bad6964c20dc74b057efa6cf2ba66e5aa8e17" exitCode=137
Jul 10 19:03:33.228459 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.228368    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"
Jul 10 19:03:33.228459 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.228408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" event={"ID":"3ab6bb30-96de-4b6b-b300-aaf51440df63","Type":"ContainerDied","Data":"a02755a92a6b961c032664388d1bad6964c20dc74b057efa6cf2ba66e5aa8e17"}
Jul 10 19:03:33.228584 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.228461    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg" event={"ID":"3ab6bb30-96de-4b6b-b300-aaf51440df63","Type":"ContainerDied","Data":"0207b7f1d7a0b85a93234cedb3e91eb609c791c944c34153876101182fa2818d"}
Jul 10 19:03:33.228584 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.228488    2565 scope.go:117] "RemoveContainer" containerID="a02755a92a6b961c032664388d1bad6964c20dc74b057efa6cf2ba66e5aa8e17"
Jul 10 19:03:33.237921 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.237905    2565 scope.go:117] "RemoveContainer" containerID="0f95271acd16cf7bcd2c42c53d644e0997ea69fb852708113d421c971b2df4cb"
Jul 10 19:03:33.245450 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.245395    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podStartSLOduration=18.766717524 podStartE2EDuration="23.245373658s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.478656134s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:03:17.181052307 +0000 UTC m=+1374.648971140" watchObservedRunningTime="2026-07-10 19:03:33.245373658 +0000 UTC m=+1390.713292494"
Jul 10 19:03:33.302413 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.302384    2565 scope.go:117] "RemoveContainer" containerID="a02755a92a6b961c032664388d1bad6964c20dc74b057efa6cf2ba66e5aa8e17"
Jul 10 19:03:33.302821 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:03:33.302705    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02755a92a6b961c032664388d1bad6964c20dc74b057efa6cf2ba66e5aa8e17\": container with ID starting with a02755a92a6b961c032664388d1bad6964c20dc74b057efa6cf2ba66e5aa8e17 not found: ID does not exist" containerID="a02755a92a6b961c032664388d1bad6964c20dc74b057efa6cf2ba66e5aa8e17"
Jul 10 19:03:33.302821 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.302742    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02755a92a6b961c032664388d1bad6964c20dc74b057efa6cf2ba66e5aa8e17"} err="failed to get container status \"a02755a92a6b961c032664388d1bad6964c20dc74b057efa6cf2ba66e5aa8e17\": rpc error: code = NotFound desc = could not find container \"a02755a92a6b961c032664388d1bad6964c20dc74b057efa6cf2ba66e5aa8e17\": container with ID starting with a02755a92a6b961c032664388d1bad6964c20dc74b057efa6cf2ba66e5aa8e17 not found: ID does not exist"
Jul 10 19:03:33.302821 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.302766    2565 scope.go:117] "RemoveContainer" containerID="0f95271acd16cf7bcd2c42c53d644e0997ea69fb852708113d421c971b2df4cb"
Jul 10 19:03:33.303043 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:03:33.303016    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f95271acd16cf7bcd2c42c53d644e0997ea69fb852708113d421c971b2df4cb\": container with ID starting with 0f95271acd16cf7bcd2c42c53d644e0997ea69fb852708113d421c971b2df4cb not found: ID does not exist" containerID="0f95271acd16cf7bcd2c42c53d644e0997ea69fb852708113d421c971b2df4cb"
Jul 10 19:03:33.303111 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.303050    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f95271acd16cf7bcd2c42c53d644e0997ea69fb852708113d421c971b2df4cb"} err="failed to get container status \"0f95271acd16cf7bcd2c42c53d644e0997ea69fb852708113d421c971b2df4cb\": rpc error: code = NotFound desc = could not find container \"0f95271acd16cf7bcd2c42c53d644e0997ea69fb852708113d421c971b2df4cb\": container with ID starting with 0f95271acd16cf7bcd2c42c53d644e0997ea69fb852708113d421c971b2df4cb not found: ID does not exist"
Jul 10 19:03:33.378071 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.378035    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-kserve-provision-location\") pod \"3ab6bb30-96de-4b6b-b300-aaf51440df63\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") "
Jul 10 19:03:33.378280 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.378106    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-tmp-dir\") pod \"3ab6bb30-96de-4b6b-b300-aaf51440df63\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") "
Jul 10 19:03:33.378280 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.378138    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-dshm\") pod \"3ab6bb30-96de-4b6b-b300-aaf51440df63\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") "
Jul 10 19:03:33.378280 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.378195    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrsmt\" (UniqueName: \"kubernetes.io/projected/3ab6bb30-96de-4b6b-b300-aaf51440df63-kube-api-access-lrsmt\") pod \"3ab6bb30-96de-4b6b-b300-aaf51440df63\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") "
Jul 10 19:03:33.378280 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.378222    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ab6bb30-96de-4b6b-b300-aaf51440df63-tls-certs\") pod \"3ab6bb30-96de-4b6b-b300-aaf51440df63\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") "
Jul 10 19:03:33.378509 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.378286    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-model-cache\") pod \"3ab6bb30-96de-4b6b-b300-aaf51440df63\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") "
Jul 10 19:03:33.378509 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.378330    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-home\") pod \"3ab6bb30-96de-4b6b-b300-aaf51440df63\" (UID: \"3ab6bb30-96de-4b6b-b300-aaf51440df63\") "
Jul 10 19:03:33.378760 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.378730    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ab6bb30-96de-4b6b-b300-aaf51440df63" (UID: "3ab6bb30-96de-4b6b-b300-aaf51440df63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:03:33.379201 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.379132    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-home" (OuterVolumeSpecName: "home") pod "3ab6bb30-96de-4b6b-b300-aaf51440df63" (UID: "3ab6bb30-96de-4b6b-b300-aaf51440df63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:03:33.380975 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.380943    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ab6bb30-96de-4b6b-b300-aaf51440df63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ab6bb30-96de-4b6b-b300-aaf51440df63" (UID: "3ab6bb30-96de-4b6b-b300-aaf51440df63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:03:33.381066 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.380983    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-dshm" (OuterVolumeSpecName: "dshm") pod "3ab6bb30-96de-4b6b-b300-aaf51440df63" (UID: "3ab6bb30-96de-4b6b-b300-aaf51440df63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:03:33.383248 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.383228    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab6bb30-96de-4b6b-b300-aaf51440df63-kube-api-access-lrsmt" (OuterVolumeSpecName: "kube-api-access-lrsmt") pod "3ab6bb30-96de-4b6b-b300-aaf51440df63" (UID: "3ab6bb30-96de-4b6b-b300-aaf51440df63"). InnerVolumeSpecName "kube-api-access-lrsmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:03:33.397967 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.397903    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ab6bb30-96de-4b6b-b300-aaf51440df63" (UID: "3ab6bb30-96de-4b6b-b300-aaf51440df63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:03:33.436883 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.436851    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ab6bb30-96de-4b6b-b300-aaf51440df63" (UID: "3ab6bb30-96de-4b6b-b300-aaf51440df63"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:03:33.479751 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.479724    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ab6bb30-96de-4b6b-b300-aaf51440df63-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:03:33.479751 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.479750    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:03:33.479898 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.479759    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:03:33.479898 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.479768    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:03:33.479898 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.479779    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:03:33.479898 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.479790    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ab6bb30-96de-4b6b-b300-aaf51440df63-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:03:33.479898 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.479798    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrsmt\" (UniqueName: \"kubernetes.io/projected/3ab6bb30-96de-4b6b-b300-aaf51440df63-kube-api-access-lrsmt\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:03:33.559584 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.554502    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"]
Jul 10 19:03:33.563068 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:33.563045    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-fb9546c6d-zzrhg"]
Jul 10 19:03:35.095260 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:35.095224    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" path="/var/lib/kubelet/pods/3ab6bb30-96de-4b6b-b300-aaf51440df63/volumes"
Jul 10 19:03:41.029463 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:41.029414    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 10 19:03:42.253818 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:42.253768    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:03:42.283103 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:42.283055    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:03:51.029185 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:51.029113    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 10 19:03:52.253585 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:52.253532    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:03:52.283337 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:03:52.283294    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:04:01.029250 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:01.029208    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 10 19:04:02.254267 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:02.254222    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:04:02.282765 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:02.282731    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:04:11.028650 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:11.028598    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 10 19:04:12.254059 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:12.254009    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:04:12.283134 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:12.283095    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:04:21.028744 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:21.028654    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 10 19:04:22.254045 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:22.253984    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:04:22.283359 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:22.283317    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:04:31.028668 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:31.028624    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 10 19:04:32.253583 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:32.253524    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:04:32.283534 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:32.283493    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:04:41.028616 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:41.028572    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 10 19:04:42.254238 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:42.254177    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:04:42.283699 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:42.283656    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:04:51.029325 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:51.029275    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 10 19:04:52.253639 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:52.253575    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:04:52.283663 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:04:52.283618    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:05:01.029414 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:05:01.029376    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 10 19:05:02.254189 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:05:02.254130    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 10 19:05:02.282934 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:05:02.282896    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 10 19:05:11.028913 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:05:11.028873    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 10 19:05:12.264046 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:05:12.264019    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:05:12.281370 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:05:12.281342    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:05:12.292086 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:05:12.292057    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:05:12.304348 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:05:12.304321    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:05:21.028831 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:05:21.028789    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 10 19:05:31.037999 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:05:31.037968    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:05:31.045299 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:05:31.045281    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:16:55.422483 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:55.422405    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"]
Jul 10 19:16:55.424989 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:55.422635    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main" containerID="cri-o://670922b34ae81f875cb96addb5a40b9baa37d87794d6847860fdbc2d0842d3ac" gracePeriod=30
Jul 10 19:16:55.427644 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:55.427610    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"]
Jul 10 19:16:55.428240 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:55.428183    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main" containerID="cri-o://9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5" gracePeriod=30
Jul 10 19:16:58.623879 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.623812    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"]
Jul 10 19:16:58.624410 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.624313    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="main"
Jul 10 19:16:58.624410 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.624332    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="main"
Jul 10 19:16:58.624410 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.624360    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="storage-initializer"
Jul 10 19:16:58.624410 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.624369    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="storage-initializer"
Jul 10 19:16:58.624618 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.624450    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ab6bb30-96de-4b6b-b300-aaf51440df63" containerName="main"
Jul 10 19:16:58.627521 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.627502    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.630245 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.630222    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\""
Jul 10 19:16:58.638521 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.638498    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"]
Jul 10 19:16:58.789528 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.789492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.789713 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.789549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.789713 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.789607    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.789713 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.789627    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.789818 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.789707    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff2cs\" (UniqueName: \"kubernetes.io/projected/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-kube-api-access-ff2cs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.789818 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.789745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.789818 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.789767    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.891183 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.891067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.891183 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.891135    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.891422 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.891216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.891422 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.891242    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.891422 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.891279    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff2cs\" (UniqueName: \"kubernetes.io/projected/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-kube-api-access-ff2cs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.891422 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.891311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.891422 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.891334    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.891712 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.891562    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.891712 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.891579    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.891712 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.891661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.891866 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.891799    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.893604 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.893574    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.893745 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.893729    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.899410 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.899390    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff2cs\" (UniqueName: \"kubernetes.io/projected/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-kube-api-access-ff2cs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:58.938162 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:58.938124    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:16:59.077795 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:59.077764    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"]
Jul 10 19:16:59.080440 ip-10-0-132-91 kubenswrapper[2565]: W0710 19:16:59.080412    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1571e38c_c6f2_40a4_a5e8_a29d9ab9c919.slice/crio-edbdbdcf98d0215871e333028e97e85730e6080d0d7a605dc2ca731fbc313d14 WatchSource:0}: Error finding container edbdbdcf98d0215871e333028e97e85730e6080d0d7a605dc2ca731fbc313d14: Status 404 returned error can't find the container with id edbdbdcf98d0215871e333028e97e85730e6080d0d7a605dc2ca731fbc313d14
Jul 10 19:16:59.760356 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:59.760320    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" event={"ID":"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919","Type":"ContainerStarted","Data":"a0331ba6692b4a1b6ef7314effbe385e2b4232aa55d4ef9e779d5af293e70744"}
Jul 10 19:16:59.760781 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:16:59.760384    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" event={"ID":"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919","Type":"ContainerStarted","Data":"edbdbdcf98d0215871e333028e97e85730e6080d0d7a605dc2ca731fbc313d14"}
Jul 10 19:17:03.780479 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:03.780443    2565 generic.go:356] "Generic (PLEG): container finished" podID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerID="a0331ba6692b4a1b6ef7314effbe385e2b4232aa55d4ef9e779d5af293e70744" exitCode=0
Jul 10 19:17:03.780901 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:03.780489    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" event={"ID":"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919","Type":"ContainerDied","Data":"a0331ba6692b4a1b6ef7314effbe385e2b4232aa55d4ef9e779d5af293e70744"}
Jul 10 19:17:04.788990 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:04.788956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" event={"ID":"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919","Type":"ContainerStarted","Data":"49f2b3b0a878b85f8eb88a1b246384fd1feefe2297153e5bdbf59ac4c6a5d31b"}
Jul 10 19:17:04.812465 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:04.812415    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" podStartSLOduration=2.401257033 podStartE2EDuration="6.812396745s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.411139712s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:17:04.810185259 +0000 UTC m=+2202.278104110" watchObservedRunningTime="2026-07-10 19:17:04.812396745 +0000 UTC m=+2202.280315577"
Jul 10 19:17:08.938566 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:08.938524    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:17:08.938980 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:08.938603    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:17:08.940124 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:08.940094    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 10 19:17:18.939064 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:18.939018    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 10 19:17:25.428448 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.428393    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="llm-d-routing-sidecar" containerID="cri-o://40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f" gracePeriod=2
Jul 10 19:17:25.714455 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.714427    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:17:25.716108 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.716090    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6fd47695bb-j7qrz_f161594b-469d-4330-bb0e-ba97bd29874c/main/0.log"
Jul 10 19:17:25.717926 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.717905    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:17:25.823753 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.823718    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-tmp-dir\") pod \"f161594b-469d-4330-bb0e-ba97bd29874c\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") "
Jul 10 19:17:25.823753 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.823755    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-home\") pod \"b4c36103-2327-4049-877e-47fe04e8b9c2\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") "
Jul 10 19:17:25.824001 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.823776    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-dshm\") pod \"f161594b-469d-4330-bb0e-ba97bd29874c\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") "
Jul 10 19:17:25.824001 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.823793    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-model-cache\") pod \"b4c36103-2327-4049-877e-47fe04e8b9c2\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") "
Jul 10 19:17:25.824001 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.823828    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-dshm\") pod \"b4c36103-2327-4049-877e-47fe04e8b9c2\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") "
Jul 10 19:17:25.824001 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.823866    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-home\") pod \"f161594b-469d-4330-bb0e-ba97bd29874c\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") "
Jul 10 19:17:25.824001 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.823888    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-tmp-dir\") pod \"b4c36103-2327-4049-877e-47fe04e8b9c2\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") "
Jul 10 19:17:25.824001 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.823917    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-kserve-provision-location\") pod \"f161594b-469d-4330-bb0e-ba97bd29874c\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") "
Jul 10 19:17:25.824001 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.823944    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c36103-2327-4049-877e-47fe04e8b9c2-tls-certs\") pod \"b4c36103-2327-4049-877e-47fe04e8b9c2\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") "
Jul 10 19:17:25.824001 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.823971    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f161594b-469d-4330-bb0e-ba97bd29874c-tls-certs\") pod \"f161594b-469d-4330-bb0e-ba97bd29874c\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") "
Jul 10 19:17:25.824450 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.824019    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfkf9\" (UniqueName: \"kubernetes.io/projected/f161594b-469d-4330-bb0e-ba97bd29874c-kube-api-access-hfkf9\") pod \"f161594b-469d-4330-bb0e-ba97bd29874c\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") "
Jul 10 19:17:25.824450 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.824077    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-model-cache\") pod \"f161594b-469d-4330-bb0e-ba97bd29874c\" (UID: \"f161594b-469d-4330-bb0e-ba97bd29874c\") "
Jul 10 19:17:25.824450 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.824091    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4c36103-2327-4049-877e-47fe04e8b9c2" (UID: "b4c36103-2327-4049-877e-47fe04e8b9c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:25.824450 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.824107    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pm27\" (UniqueName: \"kubernetes.io/projected/b4c36103-2327-4049-877e-47fe04e8b9c2-kube-api-access-8pm27\") pod \"b4c36103-2327-4049-877e-47fe04e8b9c2\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") "
Jul 10 19:17:25.824450 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.824241    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-kserve-provision-location\") pod \"b4c36103-2327-4049-877e-47fe04e8b9c2\" (UID: \"b4c36103-2327-4049-877e-47fe04e8b9c2\") "
Jul 10 19:17:25.824696 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.824531    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:25.824696 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.824607    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-home" (OuterVolumeSpecName: "home") pod "f161594b-469d-4330-bb0e-ba97bd29874c" (UID: "f161594b-469d-4330-bb0e-ba97bd29874c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:25.826506 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.825637    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-home" (OuterVolumeSpecName: "home") pod "b4c36103-2327-4049-877e-47fe04e8b9c2" (UID: "b4c36103-2327-4049-877e-47fe04e8b9c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:25.826506 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.826439    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f161594b-469d-4330-bb0e-ba97bd29874c" (UID: "f161594b-469d-4330-bb0e-ba97bd29874c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:25.829118 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.829017    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c36103-2327-4049-877e-47fe04e8b9c2-kube-api-access-8pm27" (OuterVolumeSpecName: "kube-api-access-8pm27") pod "b4c36103-2327-4049-877e-47fe04e8b9c2" (UID: "b4c36103-2327-4049-877e-47fe04e8b9c2"). InnerVolumeSpecName "kube-api-access-8pm27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:17:25.829118 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.829077    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-dshm" (OuterVolumeSpecName: "dshm") pod "f161594b-469d-4330-bb0e-ba97bd29874c" (UID: "f161594b-469d-4330-bb0e-ba97bd29874c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:25.829118 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.829095    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f161594b-469d-4330-bb0e-ba97bd29874c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f161594b-469d-4330-bb0e-ba97bd29874c" (UID: "f161594b-469d-4330-bb0e-ba97bd29874c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:17:25.829740 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.829705    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f161594b-469d-4330-bb0e-ba97bd29874c-kube-api-access-hfkf9" (OuterVolumeSpecName: "kube-api-access-hfkf9") pod "f161594b-469d-4330-bb0e-ba97bd29874c" (UID: "f161594b-469d-4330-bb0e-ba97bd29874c"). InnerVolumeSpecName "kube-api-access-hfkf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:17:25.829966 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.829939    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-dshm" (OuterVolumeSpecName: "dshm") pod "b4c36103-2327-4049-877e-47fe04e8b9c2" (UID: "b4c36103-2327-4049-877e-47fe04e8b9c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:25.830058 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.830023    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c36103-2327-4049-877e-47fe04e8b9c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4c36103-2327-4049-877e-47fe04e8b9c2" (UID: "b4c36103-2327-4049-877e-47fe04e8b9c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:17:25.844099 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.844063    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f161594b-469d-4330-bb0e-ba97bd29874c" (UID: "f161594b-469d-4330-bb0e-ba97bd29874c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:25.845823 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.845794    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4c36103-2327-4049-877e-47fe04e8b9c2" (UID: "b4c36103-2327-4049-877e-47fe04e8b9c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:25.883931 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.883884    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f161594b-469d-4330-bb0e-ba97bd29874c" (UID: "f161594b-469d-4330-bb0e-ba97bd29874c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:25.888859 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.888822    2565 generic.go:356] "Generic (PLEG): container finished" podID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerID="670922b34ae81f875cb96addb5a40b9baa37d87794d6847860fdbc2d0842d3ac" exitCode=137
Jul 10 19:17:25.889030 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.888857    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" event={"ID":"b4c36103-2327-4049-877e-47fe04e8b9c2","Type":"ContainerDied","Data":"670922b34ae81f875cb96addb5a40b9baa37d87794d6847860fdbc2d0842d3ac"}
Jul 10 19:17:25.889030 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.888884    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"
Jul 10 19:17:25.889030 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.888929    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h" event={"ID":"b4c36103-2327-4049-877e-47fe04e8b9c2","Type":"ContainerDied","Data":"fb11d59eebcafc290fad1101ba99b0af1e2ef8be4d006018069cdf39086fa74a"}
Jul 10 19:17:25.889030 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.888968    2565 scope.go:117] "RemoveContainer" containerID="670922b34ae81f875cb96addb5a40b9baa37d87794d6847860fdbc2d0842d3ac"
Jul 10 19:17:25.890623 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.890593    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6fd47695bb-j7qrz_f161594b-469d-4330-bb0e-ba97bd29874c/main/0.log"
Jul 10 19:17:25.891362 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.891338    2565 generic.go:356] "Generic (PLEG): container finished" podID="f161594b-469d-4330-bb0e-ba97bd29874c" containerID="9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5" exitCode=137
Jul 10 19:17:25.891362 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.891359    2565 generic.go:356] "Generic (PLEG): container finished" podID="f161594b-469d-4330-bb0e-ba97bd29874c" containerID="40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f" exitCode=0
Jul 10 19:17:25.891496 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.891382    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" event={"ID":"f161594b-469d-4330-bb0e-ba97bd29874c","Type":"ContainerDied","Data":"9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5"}
Jul 10 19:17:25.891496 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.891407    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"
Jul 10 19:17:25.891496 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.891444    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" event={"ID":"f161594b-469d-4330-bb0e-ba97bd29874c","Type":"ContainerDied","Data":"40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f"}
Jul 10 19:17:25.891496 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.891469    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz" event={"ID":"f161594b-469d-4330-bb0e-ba97bd29874c","Type":"ContainerDied","Data":"6206b98dac24765208e2b3e711195bdba807d9ee918183921baf25041c13c107"}
Jul 10 19:17:25.892767 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.892733    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4c36103-2327-4049-877e-47fe04e8b9c2" (UID: "b4c36103-2327-4049-877e-47fe04e8b9c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:17:25.899341 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.899323    2565 scope.go:117] "RemoveContainer" containerID="15bec37aebdf93d79be559b29dcd139f2922a7642b36ff971131bbe08cfa6811"
Jul 10 19:17:25.921189 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.916806    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"]
Jul 10 19:17:25.922532 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.921851    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6fd47695bb-j7qrz"]
Jul 10 19:17:25.924905 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.924882    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:25.924905 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.924905    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:25.925045 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.924914    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:25.925045 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.924921    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:25.925045 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.924929    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:25.925045 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.924945    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:25.925045 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.924954    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:25.925045 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.924964    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c36103-2327-4049-877e-47fe04e8b9c2-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:25.925045 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.924972    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f161594b-469d-4330-bb0e-ba97bd29874c-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:25.925045 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.924980    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfkf9\" (UniqueName: \"kubernetes.io/projected/f161594b-469d-4330-bb0e-ba97bd29874c-kube-api-access-hfkf9\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:25.925045 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.924987    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f161594b-469d-4330-bb0e-ba97bd29874c-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:25.925045 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.924996    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pm27\" (UniqueName: \"kubernetes.io/projected/b4c36103-2327-4049-877e-47fe04e8b9c2-kube-api-access-8pm27\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:25.925045 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.925004    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4c36103-2327-4049-877e-47fe04e8b9c2-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:17:25.970180 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.970117    2565 scope.go:117] "RemoveContainer" containerID="670922b34ae81f875cb96addb5a40b9baa37d87794d6847860fdbc2d0842d3ac"
Jul 10 19:17:25.970608 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:17:25.970582    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670922b34ae81f875cb96addb5a40b9baa37d87794d6847860fdbc2d0842d3ac\": container with ID starting with 670922b34ae81f875cb96addb5a40b9baa37d87794d6847860fdbc2d0842d3ac not found: ID does not exist" containerID="670922b34ae81f875cb96addb5a40b9baa37d87794d6847860fdbc2d0842d3ac"
Jul 10 19:17:25.970698 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.970618    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670922b34ae81f875cb96addb5a40b9baa37d87794d6847860fdbc2d0842d3ac"} err="failed to get container status \"670922b34ae81f875cb96addb5a40b9baa37d87794d6847860fdbc2d0842d3ac\": rpc error: code = NotFound desc = could not find container \"670922b34ae81f875cb96addb5a40b9baa37d87794d6847860fdbc2d0842d3ac\": container with ID starting with 670922b34ae81f875cb96addb5a40b9baa37d87794d6847860fdbc2d0842d3ac not found: ID does not exist"
Jul 10 19:17:25.970698 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.970638    2565 scope.go:117] "RemoveContainer" containerID="15bec37aebdf93d79be559b29dcd139f2922a7642b36ff971131bbe08cfa6811"
Jul 10 19:17:25.970939 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:17:25.970922    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15bec37aebdf93d79be559b29dcd139f2922a7642b36ff971131bbe08cfa6811\": container with ID starting with 15bec37aebdf93d79be559b29dcd139f2922a7642b36ff971131bbe08cfa6811 not found: ID does not exist" containerID="15bec37aebdf93d79be559b29dcd139f2922a7642b36ff971131bbe08cfa6811"
Jul 10 19:17:25.970985 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.970943    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bec37aebdf93d79be559b29dcd139f2922a7642b36ff971131bbe08cfa6811"} err="failed to get container status \"15bec37aebdf93d79be559b29dcd139f2922a7642b36ff971131bbe08cfa6811\": rpc error: code = NotFound desc = could not find container \"15bec37aebdf93d79be559b29dcd139f2922a7642b36ff971131bbe08cfa6811\": container with ID starting with 15bec37aebdf93d79be559b29dcd139f2922a7642b36ff971131bbe08cfa6811 not found: ID does not exist"
Jul 10 19:17:25.970985 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.970956    2565 scope.go:117] "RemoveContainer" containerID="9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5"
Jul 10 19:17:25.980516 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:25.980495    2565 scope.go:117] "RemoveContainer" containerID="4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028"
Jul 10 19:17:26.046652 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.046629    2565 scope.go:117] "RemoveContainer" containerID="40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f"
Jul 10 19:17:26.054631 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.054612    2565 scope.go:117] "RemoveContainer" containerID="9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5"
Jul 10 19:17:26.054914 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:17:26.054881    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5\": container with ID starting with 9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5 not found: ID does not exist" containerID="9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5"
Jul 10 19:17:26.054995 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.054919    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5"} err="failed to get container status \"9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5\": rpc error: code = NotFound desc = could not find container \"9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5\": container with ID starting with 9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5 not found: ID does not exist"
Jul 10 19:17:26.054995 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.054948    2565 scope.go:117] "RemoveContainer" containerID="4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028"
Jul 10 19:17:26.055314 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:17:26.055295    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028\": container with ID starting with 4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028 not found: ID does not exist" containerID="4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028"
Jul 10 19:17:26.055379 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.055326    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028"} err="failed to get container status \"4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028\": rpc error: code = NotFound desc = could not find container \"4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028\": container with ID starting with 4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028 not found: ID does not exist"
Jul 10 19:17:26.055379 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.055341    2565 scope.go:117] "RemoveContainer" containerID="40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f"
Jul 10 19:17:26.055556 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:17:26.055527    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f\": container with ID starting with 40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f not found: ID does not exist" containerID="40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f"
Jul 10 19:17:26.055625 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.055556    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f"} err="failed to get container status \"40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f\": rpc error: code = NotFound desc = could not find container \"40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f\": container with ID starting with 40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f not found: ID does not exist"
Jul 10 19:17:26.055625 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.055578    2565 scope.go:117] "RemoveContainer" containerID="9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5"
Jul 10 19:17:26.055787 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.055766    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5"} err="failed to get container status \"9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5\": rpc error: code = NotFound desc = could not find container \"9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5\": container with ID starting with 9f1f0ed0577bdcb7fd80d1a340963210962bcca8bcae4e06fabe5c13a1854bd5 not found: ID does not exist"
Jul 10 19:17:26.055787 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.055784    2565 scope.go:117] "RemoveContainer" containerID="4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028"
Jul 10 19:17:26.056013 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.055998    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028"} err="failed to get container status \"4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028\": rpc error: code = NotFound desc = could not find container \"4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028\": container with ID starting with 4ce9d98490143457084cb29ed865f9db869db159aad1263a97879be9a86e4028 not found: ID does not exist"
Jul 10 19:17:26.056013 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.056013    2565 scope.go:117] "RemoveContainer" containerID="40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f"
Jul 10 19:17:26.056266 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.056251    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f"} err="failed to get container status \"40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f\": rpc error: code = NotFound desc = could not find container \"40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f\": container with ID starting with 40275afbdf64f102cdd8ea47cdefa61a0fdc96d1d30352de7c2cb2e25e8b241f not found: ID does not exist"
Jul 10 19:17:26.218166 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.218125    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"]
Jul 10 19:17:26.223289 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:26.223213    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-765c869bbf-5fs9h"]
Jul 10 19:17:27.095082 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:27.095047    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" path="/var/lib/kubelet/pods/b4c36103-2327-4049-877e-47fe04e8b9c2/volumes"
Jul 10 19:17:27.095531 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:27.095516    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" path="/var/lib/kubelet/pods/f161594b-469d-4330-bb0e-ba97bd29874c/volumes"
Jul 10 19:17:28.938711 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:28.938684    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 10 19:17:34.976469 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:34.976409    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"]
Jul 10 19:17:34.976951 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:34.976689    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main" containerID="cri-o://9f0fb9c5ccce5aae0d40476255cefb2a1acdb4523e6a8449a688ff11e393768a" gracePeriod=30
Jul 10 19:17:38.938565 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:38.938511    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 10 19:17:48.022928 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.022729    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"]
Jul 10 19:17:48.023426 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.023404    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main"
Jul 10 19:17:48.023517 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.023429    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main"
Jul 10 19:17:48.023517 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.023449    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="storage-initializer"
Jul 10 19:17:48.023517 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.023458    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="storage-initializer"
Jul 10 19:17:48.023517 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.023471    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="llm-d-routing-sidecar"
Jul 10 19:17:48.023517 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.023479    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="llm-d-routing-sidecar"
Jul 10 19:17:48.023517 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.023489    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="storage-initializer"
Jul 10 19:17:48.023517 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.023498    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="storage-initializer"
Jul 10 19:17:48.023517 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.023507    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main"
Jul 10 19:17:48.023517 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.023516    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main"
Jul 10 19:17:48.024015 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.023596    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c36103-2327-4049-877e-47fe04e8b9c2" containerName="main"
Jul 10 19:17:48.024015 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.023609    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="main"
Jul 10 19:17:48.024015 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.023620    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f161594b-469d-4330-bb0e-ba97bd29874c" containerName="llm-d-routing-sidecar"
Jul 10 19:17:48.026373 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.026353    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.029232 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.029210    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-dwb9c\""
Jul 10 19:17:48.029600 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.029583    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
Jul 10 19:17:48.032581 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.032554    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"]
Jul 10 19:17:48.035193 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.035174    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.041980 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.041951    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"]
Jul 10 19:17:48.048983 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.048950    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"]
Jul 10 19:17:48.125806 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.125772    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.125990 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.125814    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.125990 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.125882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.125990 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.125930    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.125990 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.125954    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.125990 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.125978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.126199 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.126000    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.126199 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.126020    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq29k\" (UniqueName: \"kubernetes.io/projected/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-kube-api-access-rq29k\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.126199 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.126043    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.126199 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.126062    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34f9ebfe-9ece-463f-a37e-5091e0954c56-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.126199 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.126079    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.126199 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.126141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.126199 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.126179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.126434 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.126205    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxn2h\" (UniqueName: \"kubernetes.io/projected/34f9ebfe-9ece-463f-a37e-5091e0954c56-kube-api-access-jxn2h\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.226811 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.226771    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.226811 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.226815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.227050 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.226834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.227050 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.226979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.227050 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.227239 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227062    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq29k\" (UniqueName: \"kubernetes.io/projected/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-kube-api-access-rq29k\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.227239 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.227239 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227133    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34f9ebfe-9ece-463f-a37e-5091e0954c56-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.227239 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.227239 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227210    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.227513 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.227513 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227314    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.227513 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxn2h\" (UniqueName: \"kubernetes.io/projected/34f9ebfe-9ece-463f-a37e-5091e0954c56-kube-api-access-jxn2h\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.227513 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.227513 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227376    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.227513 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227397    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.227513 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227401    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.227513 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.227907 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227645    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.227907 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227645    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.227907 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227671    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.227907 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.227716    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.229354 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.229330    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.229969 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.229942    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.230072 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.230002    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34f9ebfe-9ece-463f-a37e-5091e0954c56-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.230285 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.230267    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.236022 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.235994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq29k\" (UniqueName: \"kubernetes.io/projected/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-kube-api-access-rq29k\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.236345 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.236323    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxn2h\" (UniqueName: \"kubernetes.io/projected/34f9ebfe-9ece-463f-a37e-5091e0954c56-kube-api-access-jxn2h\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.336918 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.336816    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:48.346122 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.346092    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:48.505611 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.505579    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"]
Jul 10 19:17:48.509497 ip-10-0-132-91 kubenswrapper[2565]: W0710 19:17:48.509459    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aea6ed6_19f0_4664_a0ca_bf457a5275d9.slice/crio-af28f68d8c16c14606e3718dabc73831a8b236940e50af2be957fcd71e972ba8 WatchSource:0}: Error finding container af28f68d8c16c14606e3718dabc73831a8b236940e50af2be957fcd71e972ba8: Status 404 returned error can't find the container with id af28f68d8c16c14606e3718dabc73831a8b236940e50af2be957fcd71e972ba8
Jul 10 19:17:48.530075 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.530046    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"]
Jul 10 19:17:48.537604 ip-10-0-132-91 kubenswrapper[2565]: W0710 19:17:48.537571    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f9ebfe_9ece_463f_a37e_5091e0954c56.slice/crio-2d93ea65afb6ee061db4bdf68422beafad8a89b117035c7852435525ae799122 WatchSource:0}: Error finding container 2d93ea65afb6ee061db4bdf68422beafad8a89b117035c7852435525ae799122: Status 404 returned error can't find the container with id 2d93ea65afb6ee061db4bdf68422beafad8a89b117035c7852435525ae799122
Jul 10 19:17:48.939611 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.939496    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 10 19:17:48.999433 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.999388    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" event={"ID":"34f9ebfe-9ece-463f-a37e-5091e0954c56","Type":"ContainerStarted","Data":"96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4"}
Jul 10 19:17:48.999636 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.999465    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" event={"ID":"34f9ebfe-9ece-463f-a37e-5091e0954c56","Type":"ContainerStarted","Data":"2d93ea65afb6ee061db4bdf68422beafad8a89b117035c7852435525ae799122"}
Jul 10 19:17:48.999766 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:48.999746    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:49.001371 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:49.001344    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" event={"ID":"0aea6ed6-19f0-4664-a0ca-bf457a5275d9","Type":"ContainerStarted","Data":"638d392f0090b7eda6efe0cd05dccb1448cc334bb0e80f6aae07ad9c0cd41d0b"}
Jul 10 19:17:49.001450 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:49.001394    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" event={"ID":"0aea6ed6-19f0-4664-a0ca-bf457a5275d9","Type":"ContainerStarted","Data":"af28f68d8c16c14606e3718dabc73831a8b236940e50af2be957fcd71e972ba8"}
Jul 10 19:17:50.013771 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:50.013632    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" event={"ID":"34f9ebfe-9ece-463f-a37e-5091e0954c56","Type":"ContainerStarted","Data":"8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45"}
Jul 10 19:17:54.036759 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:54.036718    2565 generic.go:356] "Generic (PLEG): container finished" podID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerID="8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45" exitCode=0
Jul 10 19:17:54.037226 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:54.036776    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" event={"ID":"34f9ebfe-9ece-463f-a37e-5091e0954c56","Type":"ContainerDied","Data":"8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45"}
Jul 10 19:17:54.038441 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:54.038420    2565 generic.go:356] "Generic (PLEG): container finished" podID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerID="638d392f0090b7eda6efe0cd05dccb1448cc334bb0e80f6aae07ad9c0cd41d0b" exitCode=0
Jul 10 19:17:54.038529 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:54.038491    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" event={"ID":"0aea6ed6-19f0-4664-a0ca-bf457a5275d9","Type":"ContainerDied","Data":"638d392f0090b7eda6efe0cd05dccb1448cc334bb0e80f6aae07ad9c0cd41d0b"}
Jul 10 19:17:55.055189 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:55.055130    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" event={"ID":"34f9ebfe-9ece-463f-a37e-5091e0954c56","Type":"ContainerStarted","Data":"8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8"}
Jul 10 19:17:55.057560 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:55.057521    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" event={"ID":"0aea6ed6-19f0-4664-a0ca-bf457a5275d9","Type":"ContainerStarted","Data":"cf876dad2af723dd96feb12e28658885563824a4e46827cd925ceb522d3cd5c6"}
Jul 10 19:17:55.081997 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:55.081930    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podStartSLOduration=2.43890262 podStartE2EDuration="7.081909266s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.643006646s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:17:55.080892817 +0000 UTC m=+2252.548811698" watchObservedRunningTime="2026-07-10 19:17:55.081909266 +0000 UTC m=+2252.549828100"
Jul 10 19:17:55.103316 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:55.103244    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podStartSLOduration=2.6320326080000003 podStartE2EDuration="7.103222181s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.471189573s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:17:55.102258442 +0000 UTC m=+2252.570177298" watchObservedRunningTime="2026-07-10 19:17:55.103222181 +0000 UTC m=+2252.571141014"
Jul 10 19:17:58.337567 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:58.337523    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:58.339860 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:58.339824    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:58.340276 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:58.339995    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 10 19:17:58.346311 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:58.346276    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:58.346507 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:58.346491    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:17:58.347665 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:58.347636    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:17:58.353097 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:58.353080    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:17:58.939648 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:17:58.939602    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 10 19:18:05.381237 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.381212    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-76b768cb56-46p5r_f46b0492-ba9a-4580-bccc-da507c3f55ab/main/0.log"
Jul 10 19:18:05.398726 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.398696    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:18:05.475886 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.475850    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46b0492-ba9a-4580-bccc-da507c3f55ab-tls-certs\") pod \"f46b0492-ba9a-4580-bccc-da507c3f55ab\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") "
Jul 10 19:18:05.475886 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.475897    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-tmp-dir\") pod \"f46b0492-ba9a-4580-bccc-da507c3f55ab\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") "
Jul 10 19:18:05.476178 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.475931    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-kserve-provision-location\") pod \"f46b0492-ba9a-4580-bccc-da507c3f55ab\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") "
Jul 10 19:18:05.476178 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.475987    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-model-cache\") pod \"f46b0492-ba9a-4580-bccc-da507c3f55ab\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") "
Jul 10 19:18:05.476178 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.476025    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6mbj\" (UniqueName: \"kubernetes.io/projected/f46b0492-ba9a-4580-bccc-da507c3f55ab-kube-api-access-d6mbj\") pod \"f46b0492-ba9a-4580-bccc-da507c3f55ab\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") "
Jul 10 19:18:05.476178 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.476098    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-dshm\") pod \"f46b0492-ba9a-4580-bccc-da507c3f55ab\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") "
Jul 10 19:18:05.476178 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.476126    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-home\") pod \"f46b0492-ba9a-4580-bccc-da507c3f55ab\" (UID: \"f46b0492-ba9a-4580-bccc-da507c3f55ab\") "
Jul 10 19:18:05.477011 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.476976    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-home" (OuterVolumeSpecName: "home") pod "f46b0492-ba9a-4580-bccc-da507c3f55ab" (UID: "f46b0492-ba9a-4580-bccc-da507c3f55ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:18:05.478839 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.478808    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46b0492-ba9a-4580-bccc-da507c3f55ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f46b0492-ba9a-4580-bccc-da507c3f55ab" (UID: "f46b0492-ba9a-4580-bccc-da507c3f55ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:18:05.479064 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.479011    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "f46b0492-ba9a-4580-bccc-da507c3f55ab" (UID: "f46b0492-ba9a-4580-bccc-da507c3f55ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:18:05.479640 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.479608    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46b0492-ba9a-4580-bccc-da507c3f55ab-kube-api-access-d6mbj" (OuterVolumeSpecName: "kube-api-access-d6mbj") pod "f46b0492-ba9a-4580-bccc-da507c3f55ab" (UID: "f46b0492-ba9a-4580-bccc-da507c3f55ab"). InnerVolumeSpecName "kube-api-access-d6mbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:18:05.479926 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.479907    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-dshm" (OuterVolumeSpecName: "dshm") pod "f46b0492-ba9a-4580-bccc-da507c3f55ab" (UID: "f46b0492-ba9a-4580-bccc-da507c3f55ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:18:05.491638 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.491600    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f46b0492-ba9a-4580-bccc-da507c3f55ab" (UID: "f46b0492-ba9a-4580-bccc-da507c3f55ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:18:05.537791 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.537698    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f46b0492-ba9a-4580-bccc-da507c3f55ab" (UID: "f46b0492-ba9a-4580-bccc-da507c3f55ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:18:05.577608 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.577569    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f46b0492-ba9a-4580-bccc-da507c3f55ab-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:18:05.577608 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.577601    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:18:05.577608 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.577617    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:18:05.577852 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.577628    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:18:05.577852 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.577640    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6mbj\" (UniqueName: \"kubernetes.io/projected/f46b0492-ba9a-4580-bccc-da507c3f55ab-kube-api-access-d6mbj\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:18:05.577852 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.577651    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:18:05.577852 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:05.577660    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f46b0492-ba9a-4580-bccc-da507c3f55ab-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:18:06.131667 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:06.131636    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-76b768cb56-46p5r_f46b0492-ba9a-4580-bccc-da507c3f55ab/main/0.log"
Jul 10 19:18:06.132127 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:06.132097    2565 generic.go:356] "Generic (PLEG): container finished" podID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerID="9f0fb9c5ccce5aae0d40476255cefb2a1acdb4523e6a8449a688ff11e393768a" exitCode=137
Jul 10 19:18:06.132287 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:06.132186    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" event={"ID":"f46b0492-ba9a-4580-bccc-da507c3f55ab","Type":"ContainerDied","Data":"9f0fb9c5ccce5aae0d40476255cefb2a1acdb4523e6a8449a688ff11e393768a"}
Jul 10 19:18:06.132287 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:06.132224    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"
Jul 10 19:18:06.132287 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:06.132258    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r" event={"ID":"f46b0492-ba9a-4580-bccc-da507c3f55ab","Type":"ContainerDied","Data":"17d648f93a5c522e480ae0591a258cb58bd7b409a223f2c37f8bc92ff0b403ce"}
Jul 10 19:18:06.132449 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:06.132292    2565 scope.go:117] "RemoveContainer" containerID="9f0fb9c5ccce5aae0d40476255cefb2a1acdb4523e6a8449a688ff11e393768a"
Jul 10 19:18:06.146123 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:06.146096    2565 scope.go:117] "RemoveContainer" containerID="c1fc6ee1975257b48452c61510c035023e0aa6f2c6d08fdb89b602a1fc915832"
Jul 10 19:18:06.157809 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:06.157678    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"]
Jul 10 19:18:06.157894 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:06.157874    2565 scope.go:117] "RemoveContainer" containerID="9f0fb9c5ccce5aae0d40476255cefb2a1acdb4523e6a8449a688ff11e393768a"
Jul 10 19:18:06.161114 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:18:06.158391    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0fb9c5ccce5aae0d40476255cefb2a1acdb4523e6a8449a688ff11e393768a\": container with ID starting with 9f0fb9c5ccce5aae0d40476255cefb2a1acdb4523e6a8449a688ff11e393768a not found: ID does not exist" containerID="9f0fb9c5ccce5aae0d40476255cefb2a1acdb4523e6a8449a688ff11e393768a"
Jul 10 19:18:06.161114 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:06.158433    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0fb9c5ccce5aae0d40476255cefb2a1acdb4523e6a8449a688ff11e393768a"} err="failed to get container status \"9f0fb9c5ccce5aae0d40476255cefb2a1acdb4523e6a8449a688ff11e393768a\": rpc error: code = NotFound desc = could not find container \"9f0fb9c5ccce5aae0d40476255cefb2a1acdb4523e6a8449a688ff11e393768a\": container with ID starting with 9f0fb9c5ccce5aae0d40476255cefb2a1acdb4523e6a8449a688ff11e393768a not found: ID does not exist"
Jul 10 19:18:06.161114 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:06.158473    2565 scope.go:117] "RemoveContainer" containerID="c1fc6ee1975257b48452c61510c035023e0aa6f2c6d08fdb89b602a1fc915832"
Jul 10 19:18:06.161114 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:18:06.158891    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1fc6ee1975257b48452c61510c035023e0aa6f2c6d08fdb89b602a1fc915832\": container with ID starting with c1fc6ee1975257b48452c61510c035023e0aa6f2c6d08fdb89b602a1fc915832 not found: ID does not exist" containerID="c1fc6ee1975257b48452c61510c035023e0aa6f2c6d08fdb89b602a1fc915832"
Jul 10 19:18:06.161114 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:06.158938    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1fc6ee1975257b48452c61510c035023e0aa6f2c6d08fdb89b602a1fc915832"} err="failed to get container status \"c1fc6ee1975257b48452c61510c035023e0aa6f2c6d08fdb89b602a1fc915832\": rpc error: code = NotFound desc = could not find container \"c1fc6ee1975257b48452c61510c035023e0aa6f2c6d08fdb89b602a1fc915832\": container with ID starting with c1fc6ee1975257b48452c61510c035023e0aa6f2c6d08fdb89b602a1fc915832 not found: ID does not exist"
Jul 10 19:18:06.161114 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:06.160970    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76b768cb56-46p5r"]
Jul 10 19:18:07.097165 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:07.097111    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" path="/var/lib/kubelet/pods/f46b0492-ba9a-4580-bccc-da507c3f55ab/volumes"
Jul 10 19:18:08.337570 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:08.337521    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 10 19:18:08.347420 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:08.347375    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:18:08.938847 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:08.938786    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 10 19:18:18.337746 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:18.337686    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 10 19:18:18.346894 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:18.346853    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:18:18.939627 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:18.939572    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 10 19:18:28.338053 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:28.338008    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 10 19:18:28.346981 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:28.346939    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:18:28.939599 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:28.939554    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 10 19:18:38.338170 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:38.338100    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 10 19:18:38.347440 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:38.347395    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:18:38.938806 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:38.938764    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Jul 10 19:18:48.337602 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:48.337549    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 10 19:18:48.347222 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:48.347182    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:18:48.949490 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:48.949458    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:18:48.957634 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:48.957610    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:18:58.337560 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:58.337512    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 10 19:18:58.346600 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:58.346544    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:18:59.535654 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:59.535622    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"]
Jul 10 19:18:59.536111 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:18:59.535845    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="main" containerID="cri-o://49f2b3b0a878b85f8eb88a1b246384fd1feefe2297153e5bdbf59ac4c6a5d31b" gracePeriod=30
Jul 10 19:19:08.337597 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:08.337550    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 10 19:19:08.346733 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:08.346690    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:19:13.643392 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.643342    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 10 19:19:13.643892 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.643809    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="storage-initializer"
Jul 10 19:19:13.643892 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.643830    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="storage-initializer"
Jul 10 19:19:13.643892 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.643847    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main"
Jul 10 19:19:13.643892 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.643857    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main"
Jul 10 19:19:13.644068 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.643954    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f46b0492-ba9a-4580-bccc-da507c3f55ab" containerName="main"
Jul 10 19:19:13.646446 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.646425    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.649088 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.649062    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-762zd\""
Jul 10 19:19:13.649219 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.649109    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
Jul 10 19:19:13.657162 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.657116    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 10 19:19:13.717438 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.717405    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.717635 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.717445    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c35f1a-1af5-4d58-897c-f74d09514f26-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.717635 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.717527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.717635 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.717565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhbq5\" (UniqueName: \"kubernetes.io/projected/70c35f1a-1af5-4d58-897c-f74d09514f26-kube-api-access-bhbq5\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.717635 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.717623    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.717805 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.717645    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.717805 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.717680    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.818935 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.818891    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.819120 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.818946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c35f1a-1af5-4d58-897c-f74d09514f26-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.819120 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.819007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.819120 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.819035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhbq5\" (UniqueName: \"kubernetes.io/projected/70c35f1a-1af5-4d58-897c-f74d09514f26-kube-api-access-bhbq5\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.819120 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.819085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.819120 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.819116    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.819424 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.819195    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.819424 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.819387    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.819538 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.819514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.819633 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.819608    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.819694 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.819655    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.821349 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.821322    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.821822 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.821800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c35f1a-1af5-4d58-897c-f74d09514f26-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.833871 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.833848    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhbq5\" (UniqueName: \"kubernetes.io/projected/70c35f1a-1af5-4d58-897c-f74d09514f26-kube-api-access-bhbq5\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:13.959702 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:13.959617    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:14.097585 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:14.097516    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 10 19:19:14.100086 ip-10-0-132-91 kubenswrapper[2565]: W0710 19:19:14.100047    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c35f1a_1af5_4d58_897c_f74d09514f26.slice/crio-540faafb82021419e4b8d126accf7aa4800953284241d1c5d90849398b2f0cbf WatchSource:0}: Error finding container 540faafb82021419e4b8d126accf7aa4800953284241d1c5d90849398b2f0cbf: Status 404 returned error can't find the container with id 540faafb82021419e4b8d126accf7aa4800953284241d1c5d90849398b2f0cbf
Jul 10 19:19:14.503519 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:14.503436    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"70c35f1a-1af5-4d58-897c-f74d09514f26","Type":"ContainerStarted","Data":"fca178949bb037af18b8517f52b03af5ed92b41e2db8869a7957a4f971f1e915"}
Jul 10 19:19:14.503519 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:14.503505    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"70c35f1a-1af5-4d58-897c-f74d09514f26","Type":"ContainerStarted","Data":"540faafb82021419e4b8d126accf7aa4800953284241d1c5d90849398b2f0cbf"}
Jul 10 19:19:18.337786 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:18.337735    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 10 19:19:18.346973 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:18.346935    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:19:19.529393 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:19.529359    2565 generic.go:356] "Generic (PLEG): container finished" podID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerID="fca178949bb037af18b8517f52b03af5ed92b41e2db8869a7957a4f971f1e915" exitCode=0
Jul 10 19:19:19.529766 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:19.529439    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"70c35f1a-1af5-4d58-897c-f74d09514f26","Type":"ContainerDied","Data":"fca178949bb037af18b8517f52b03af5ed92b41e2db8869a7957a4f971f1e915"}
Jul 10 19:19:20.537046 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:20.537013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"70c35f1a-1af5-4d58-897c-f74d09514f26","Type":"ContainerStarted","Data":"227a15a8d829a2d0f1665d5c4025f612ef701d1fc6e08b58b36dd31b6ac2cbb3"}
Jul 10 19:19:20.560849 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:20.560784    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=2.8541739550000003 podStartE2EDuration="7.560761132s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.706587177s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:19:20.559207451 +0000 UTC m=+2338.027126303" watchObservedRunningTime="2026-07-10 19:19:20.560761132 +0000 UTC m=+2338.028679967"
Jul 10 19:19:23.960267 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:23.960222    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:23.961847 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:23.961813    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 10 19:19:28.337817 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:28.337769    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 10 19:19:28.346472 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:28.346431    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:19:29.935673 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.935646    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg_1571e38c-c6f2-40a4-a5e8-a29d9ab9c919/main/0.log"
Jul 10 19:19:29.956271 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.956244    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:19:29.966027 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.966008    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff2cs\" (UniqueName: \"kubernetes.io/projected/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-kube-api-access-ff2cs\") pod \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") "
Jul 10 19:19:29.966104 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.966047    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-kserve-provision-location\") pod \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") "
Jul 10 19:19:29.966293 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.966270    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-tmp-dir\") pod \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") "
Jul 10 19:19:29.966371 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.966321    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-home\") pod \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") "
Jul 10 19:19:29.966371 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.966361    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-model-cache\") pod \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") "
Jul 10 19:19:29.966480 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.966387    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-tls-certs\") pod \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") "
Jul 10 19:19:29.966480 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.966433    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-dshm\") pod \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\" (UID: \"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919\") "
Jul 10 19:19:29.967990 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.967486    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-model-cache" (OuterVolumeSpecName: "model-cache") pod "1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" (UID: "1571e38c-c6f2-40a4-a5e8-a29d9ab9c919"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:19:29.967990 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.967952    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-home" (OuterVolumeSpecName: "home") pod "1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" (UID: "1571e38c-c6f2-40a4-a5e8-a29d9ab9c919"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:19:29.969449 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.969333    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" (UID: "1571e38c-c6f2-40a4-a5e8-a29d9ab9c919"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:19:29.969449 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.969414    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-kube-api-access-ff2cs" (OuterVolumeSpecName: "kube-api-access-ff2cs") pod "1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" (UID: "1571e38c-c6f2-40a4-a5e8-a29d9ab9c919"). InnerVolumeSpecName "kube-api-access-ff2cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:19:29.969692 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.969667    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-dshm" (OuterVolumeSpecName: "dshm") pod "1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" (UID: "1571e38c-c6f2-40a4-a5e8-a29d9ab9c919"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:19:29.981406 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:29.980568    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" (UID: "1571e38c-c6f2-40a4-a5e8-a29d9ab9c919"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:19:30.043162 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.043097    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" (UID: "1571e38c-c6f2-40a4-a5e8-a29d9ab9c919"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:19:30.067483 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.067451    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:19:30.067483 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.067480    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:19:30.067483 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.067489    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:19:30.067673 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.067498    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff2cs\" (UniqueName: \"kubernetes.io/projected/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-kube-api-access-ff2cs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:19:30.067673 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.067507    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:19:30.067673 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.067515    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:19:30.067673 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.067524    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:19:30.587037 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.587010    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg_1571e38c-c6f2-40a4-a5e8-a29d9ab9c919/main/0.log"
Jul 10 19:19:30.587499 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.587471    2565 generic.go:356] "Generic (PLEG): container finished" podID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerID="49f2b3b0a878b85f8eb88a1b246384fd1feefe2297153e5bdbf59ac4c6a5d31b" exitCode=137
Jul 10 19:19:30.587640 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.587534    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"
Jul 10 19:19:30.587640 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.587556    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" event={"ID":"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919","Type":"ContainerDied","Data":"49f2b3b0a878b85f8eb88a1b246384fd1feefe2297153e5bdbf59ac4c6a5d31b"}
Jul 10 19:19:30.587640 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.587626    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg" event={"ID":"1571e38c-c6f2-40a4-a5e8-a29d9ab9c919","Type":"ContainerDied","Data":"edbdbdcf98d0215871e333028e97e85730e6080d0d7a605dc2ca731fbc313d14"}
Jul 10 19:19:30.587801 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.587655    2565 scope.go:117] "RemoveContainer" containerID="49f2b3b0a878b85f8eb88a1b246384fd1feefe2297153e5bdbf59ac4c6a5d31b"
Jul 10 19:19:30.598360 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.598339    2565 scope.go:117] "RemoveContainer" containerID="a0331ba6692b4a1b6ef7314effbe385e2b4232aa55d4ef9e779d5af293e70744"
Jul 10 19:19:30.608743 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.608725    2565 scope.go:117] "RemoveContainer" containerID="49f2b3b0a878b85f8eb88a1b246384fd1feefe2297153e5bdbf59ac4c6a5d31b"
Jul 10 19:19:30.609057 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:19:30.609038    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f2b3b0a878b85f8eb88a1b246384fd1feefe2297153e5bdbf59ac4c6a5d31b\": container with ID starting with 49f2b3b0a878b85f8eb88a1b246384fd1feefe2297153e5bdbf59ac4c6a5d31b not found: ID does not exist" containerID="49f2b3b0a878b85f8eb88a1b246384fd1feefe2297153e5bdbf59ac4c6a5d31b"
Jul 10 19:19:30.609157 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.609070    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f2b3b0a878b85f8eb88a1b246384fd1feefe2297153e5bdbf59ac4c6a5d31b"} err="failed to get container status \"49f2b3b0a878b85f8eb88a1b246384fd1feefe2297153e5bdbf59ac4c6a5d31b\": rpc error: code = NotFound desc = could not find container \"49f2b3b0a878b85f8eb88a1b246384fd1feefe2297153e5bdbf59ac4c6a5d31b\": container with ID starting with 49f2b3b0a878b85f8eb88a1b246384fd1feefe2297153e5bdbf59ac4c6a5d31b not found: ID does not exist"
Jul 10 19:19:30.609157 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.609096    2565 scope.go:117] "RemoveContainer" containerID="a0331ba6692b4a1b6ef7314effbe385e2b4232aa55d4ef9e779d5af293e70744"
Jul 10 19:19:30.609387 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:19:30.609368    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0331ba6692b4a1b6ef7314effbe385e2b4232aa55d4ef9e779d5af293e70744\": container with ID starting with a0331ba6692b4a1b6ef7314effbe385e2b4232aa55d4ef9e779d5af293e70744 not found: ID does not exist" containerID="a0331ba6692b4a1b6ef7314effbe385e2b4232aa55d4ef9e779d5af293e70744"
Jul 10 19:19:30.609438 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.609393    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0331ba6692b4a1b6ef7314effbe385e2b4232aa55d4ef9e779d5af293e70744"} err="failed to get container status \"a0331ba6692b4a1b6ef7314effbe385e2b4232aa55d4ef9e779d5af293e70744\": rpc error: code = NotFound desc = could not find container \"a0331ba6692b4a1b6ef7314effbe385e2b4232aa55d4ef9e779d5af293e70744\": container with ID starting with a0331ba6692b4a1b6ef7314effbe385e2b4232aa55d4ef9e779d5af293e70744 not found: ID does not exist"
Jul 10 19:19:30.614966 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.614942    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"]
Jul 10 19:19:30.618654 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:30.618631    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b5946b9b4vkxrg"]
Jul 10 19:19:31.096068 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:31.096035    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" path="/var/lib/kubelet/pods/1571e38c-c6f2-40a4-a5e8-a29d9ab9c919/volumes"
Jul 10 19:19:33.960860 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:33.960815    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 10 19:19:38.337433 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:38.337387    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 10 19:19:38.347220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:38.347177    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:19:43.959912 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:43.959869    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:19:43.960563 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:43.960350    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 10 19:19:48.337411 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:48.337346    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 10 19:19:48.347315 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:48.347249    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:19:53.960311 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:53.960262    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 10 19:19:58.337429 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:58.337379    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 10 19:19:58.346518 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:19:58.346478    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:20:03.960433 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:03.960385    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 10 19:20:08.338204 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:08.338137    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8001/health\": dial tcp 10.134.0.39:8001: connect: connection refused"
Jul 10 19:20:08.346902 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:08.346862    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:20:13.961074 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:13.961035    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 10 19:20:18.347455 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:18.347405    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 10 19:20:18.352314 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:18.352282    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:20:18.369702 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:18.369671    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:20:23.960801 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:23.960754    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 10 19:20:28.359197 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:28.359167    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:20:28.371450 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:28.371424    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:20:33.960342 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:33.960301    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 10 19:20:43.960279 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:43.960212    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 10 19:20:47.080550 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:47.080462    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"]
Jul 10 19:20:47.081006 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:47.080862    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main" containerID="cri-o://8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8" gracePeriod=30
Jul 10 19:20:47.089465 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:47.089432    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"]
Jul 10 19:20:47.089929 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:47.089893    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main" containerID="cri-o://cf876dad2af723dd96feb12e28658885563824a4e46827cd925ceb522d3cd5c6" gracePeriod=30
Jul 10 19:20:53.960864 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:20:53.960814    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 10 19:21:03.960823 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:03.960773    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 10 19:21:13.970250 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:13.970219    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:21:13.977654 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:13.977631    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:21:17.081201 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.081153    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="llm-d-routing-sidecar" containerID="cri-o://96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4" gracePeriod=2
Jul 10 19:21:17.374461 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.374428    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm_34f9ebfe-9ece-463f-a37e-5091e0954c56/main/0.log"
Jul 10 19:21:17.377955 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.377929    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:21:17.380411 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.380386    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:21:17.422757 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.422731    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-model-cache\") pod \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") "
Jul 10 19:21:17.422757 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.422763    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-kserve-provision-location\") pod \"34f9ebfe-9ece-463f-a37e-5091e0954c56\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") "
Jul 10 19:21:17.423014 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.422799    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxn2h\" (UniqueName: \"kubernetes.io/projected/34f9ebfe-9ece-463f-a37e-5091e0954c56-kube-api-access-jxn2h\") pod \"34f9ebfe-9ece-463f-a37e-5091e0954c56\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") "
Jul 10 19:21:17.423014 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.422824    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-model-cache\") pod \"34f9ebfe-9ece-463f-a37e-5091e0954c56\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") "
Jul 10 19:21:17.423014 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.422862    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-dshm\") pod \"34f9ebfe-9ece-463f-a37e-5091e0954c56\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") "
Jul 10 19:21:17.423014 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.422889    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-tmp-dir\") pod \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") "
Jul 10 19:21:17.423014 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.422932    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-home\") pod \"34f9ebfe-9ece-463f-a37e-5091e0954c56\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") "
Jul 10 19:21:17.423014 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.422957    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rq29k\" (UniqueName: \"kubernetes.io/projected/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-kube-api-access-rq29k\") pod \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") "
Jul 10 19:21:17.423014 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.422983    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-kserve-provision-location\") pod \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") "
Jul 10 19:21:17.423014 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.423008    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-tmp-dir\") pod \"34f9ebfe-9ece-463f-a37e-5091e0954c56\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") "
Jul 10 19:21:17.423461 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.423039    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-home\") pod \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") "
Jul 10 19:21:17.423461 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.423045    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "0aea6ed6-19f0-4664-a0ca-bf457a5275d9" (UID: "0aea6ed6-19f0-4664-a0ca-bf457a5275d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:17.423461 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.423075    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-tls-certs\") pod \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") "
Jul 10 19:21:17.423461 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.423120    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34f9ebfe-9ece-463f-a37e-5091e0954c56-tls-certs\") pod \"34f9ebfe-9ece-463f-a37e-5091e0954c56\" (UID: \"34f9ebfe-9ece-463f-a37e-5091e0954c56\") "
Jul 10 19:21:17.423461 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.423176    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-dshm\") pod \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\" (UID: \"0aea6ed6-19f0-4664-a0ca-bf457a5275d9\") "
Jul 10 19:21:17.423461 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.423425    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:17.423773 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.423675    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-home" (OuterVolumeSpecName: "home") pod "34f9ebfe-9ece-463f-a37e-5091e0954c56" (UID: "34f9ebfe-9ece-463f-a37e-5091e0954c56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:17.425356 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.425325    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-model-cache" (OuterVolumeSpecName: "model-cache") pod "34f9ebfe-9ece-463f-a37e-5091e0954c56" (UID: "34f9ebfe-9ece-463f-a37e-5091e0954c56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:17.425822 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.425796    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-dshm" (OuterVolumeSpecName: "dshm") pod "0aea6ed6-19f0-4664-a0ca-bf457a5275d9" (UID: "0aea6ed6-19f0-4664-a0ca-bf457a5275d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:17.426411 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.426390    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-home" (OuterVolumeSpecName: "home") pod "0aea6ed6-19f0-4664-a0ca-bf457a5275d9" (UID: "0aea6ed6-19f0-4664-a0ca-bf457a5275d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:17.434935 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.434891    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-dshm" (OuterVolumeSpecName: "dshm") pod "34f9ebfe-9ece-463f-a37e-5091e0954c56" (UID: "34f9ebfe-9ece-463f-a37e-5091e0954c56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:17.435074 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.435008    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f9ebfe-9ece-463f-a37e-5091e0954c56-kube-api-access-jxn2h" (OuterVolumeSpecName: "kube-api-access-jxn2h") pod "34f9ebfe-9ece-463f-a37e-5091e0954c56" (UID: "34f9ebfe-9ece-463f-a37e-5091e0954c56"). InnerVolumeSpecName "kube-api-access-jxn2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:21:17.438197 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.436113    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-kube-api-access-rq29k" (OuterVolumeSpecName: "kube-api-access-rq29k") pod "0aea6ed6-19f0-4664-a0ca-bf457a5275d9" (UID: "0aea6ed6-19f0-4664-a0ca-bf457a5275d9"). InnerVolumeSpecName "kube-api-access-rq29k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:21:17.450941 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.450900    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0aea6ed6-19f0-4664-a0ca-bf457a5275d9" (UID: "0aea6ed6-19f0-4664-a0ca-bf457a5275d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:21:17.451454 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.451418    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34f9ebfe-9ece-463f-a37e-5091e0954c56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34f9ebfe-9ece-463f-a37e-5091e0954c56" (UID: "34f9ebfe-9ece-463f-a37e-5091e0954c56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:21:17.456570 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.456540    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34f9ebfe-9ece-463f-a37e-5091e0954c56" (UID: "34f9ebfe-9ece-463f-a37e-5091e0954c56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:17.458160 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.458121    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0aea6ed6-19f0-4664-a0ca-bf457a5275d9" (UID: "0aea6ed6-19f0-4664-a0ca-bf457a5275d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:17.492197 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.492159    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0aea6ed6-19f0-4664-a0ca-bf457a5275d9" (UID: "0aea6ed6-19f0-4664-a0ca-bf457a5275d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:17.504137 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.504092    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34f9ebfe-9ece-463f-a37e-5091e0954c56" (UID: "34f9ebfe-9ece-463f-a37e-5091e0954c56"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:17.524735 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.524706    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:17.524857 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.524758    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:17.524857 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.524767    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:17.524857 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.524776    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq29k\" (UniqueName: \"kubernetes.io/projected/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-kube-api-access-rq29k\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:17.524857 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.524787    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:17.524857 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.524795    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:17.524857 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.524802    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:17.524857 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.524810    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:17.524857 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.524818    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34f9ebfe-9ece-463f-a37e-5091e0954c56-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:17.524857 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.524825    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aea6ed6-19f0-4664-a0ca-bf457a5275d9-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:17.524857 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.524832    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:17.524857 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.524840    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxn2h\" (UniqueName: \"kubernetes.io/projected/34f9ebfe-9ece-463f-a37e-5091e0954c56-kube-api-access-jxn2h\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:17.524857 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:17.524849    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34f9ebfe-9ece-463f-a37e-5091e0954c56-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:18.111691 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.111659    2565 generic.go:356] "Generic (PLEG): container finished" podID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerID="cf876dad2af723dd96feb12e28658885563824a4e46827cd925ceb522d3cd5c6" exitCode=137
Jul 10 19:21:18.112092 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.111720    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"
Jul 10 19:21:18.112092 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.111742    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" event={"ID":"0aea6ed6-19f0-4664-a0ca-bf457a5275d9","Type":"ContainerDied","Data":"cf876dad2af723dd96feb12e28658885563824a4e46827cd925ceb522d3cd5c6"}
Jul 10 19:21:18.112092 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.111803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w" event={"ID":"0aea6ed6-19f0-4664-a0ca-bf457a5275d9","Type":"ContainerDied","Data":"af28f68d8c16c14606e3718dabc73831a8b236940e50af2be957fcd71e972ba8"}
Jul 10 19:21:18.112092 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.111832    2565 scope.go:117] "RemoveContainer" containerID="cf876dad2af723dd96feb12e28658885563824a4e46827cd925ceb522d3cd5c6"
Jul 10 19:21:18.113480 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.113460    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm_34f9ebfe-9ece-463f-a37e-5091e0954c56/main/0.log"
Jul 10 19:21:18.114254 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.114229    2565 generic.go:356] "Generic (PLEG): container finished" podID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerID="8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8" exitCode=137
Jul 10 19:21:18.114254 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.114253    2565 generic.go:356] "Generic (PLEG): container finished" podID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerID="96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4" exitCode=0
Jul 10 19:21:18.114418 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.114284    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" event={"ID":"34f9ebfe-9ece-463f-a37e-5091e0954c56","Type":"ContainerDied","Data":"8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8"}
Jul 10 19:21:18.114418 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.114329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" event={"ID":"34f9ebfe-9ece-463f-a37e-5091e0954c56","Type":"ContainerDied","Data":"96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4"}
Jul 10 19:21:18.114418 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.114354    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm" event={"ID":"34f9ebfe-9ece-463f-a37e-5091e0954c56","Type":"ContainerDied","Data":"2d93ea65afb6ee061db4bdf68422beafad8a89b117035c7852435525ae799122"}
Jul 10 19:21:18.114418 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.114291    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"
Jul 10 19:21:18.121287 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.121247    2565 scope.go:117] "RemoveContainer" containerID="638d392f0090b7eda6efe0cd05dccb1448cc334bb0e80f6aae07ad9c0cd41d0b"
Jul 10 19:21:18.132090 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.132071    2565 scope.go:117] "RemoveContainer" containerID="cf876dad2af723dd96feb12e28658885563824a4e46827cd925ceb522d3cd5c6"
Jul 10 19:21:18.132404 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:21:18.132376    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf876dad2af723dd96feb12e28658885563824a4e46827cd925ceb522d3cd5c6\": container with ID starting with cf876dad2af723dd96feb12e28658885563824a4e46827cd925ceb522d3cd5c6 not found: ID does not exist" containerID="cf876dad2af723dd96feb12e28658885563824a4e46827cd925ceb522d3cd5c6"
Jul 10 19:21:18.132500 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.132410    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf876dad2af723dd96feb12e28658885563824a4e46827cd925ceb522d3cd5c6"} err="failed to get container status \"cf876dad2af723dd96feb12e28658885563824a4e46827cd925ceb522d3cd5c6\": rpc error: code = NotFound desc = could not find container \"cf876dad2af723dd96feb12e28658885563824a4e46827cd925ceb522d3cd5c6\": container with ID starting with cf876dad2af723dd96feb12e28658885563824a4e46827cd925ceb522d3cd5c6 not found: ID does not exist"
Jul 10 19:21:18.132500 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.132427    2565 scope.go:117] "RemoveContainer" containerID="638d392f0090b7eda6efe0cd05dccb1448cc334bb0e80f6aae07ad9c0cd41d0b"
Jul 10 19:21:18.132699 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:21:18.132670    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638d392f0090b7eda6efe0cd05dccb1448cc334bb0e80f6aae07ad9c0cd41d0b\": container with ID starting with 638d392f0090b7eda6efe0cd05dccb1448cc334bb0e80f6aae07ad9c0cd41d0b not found: ID does not exist" containerID="638d392f0090b7eda6efe0cd05dccb1448cc334bb0e80f6aae07ad9c0cd41d0b"
Jul 10 19:21:18.132764 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.132703    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638d392f0090b7eda6efe0cd05dccb1448cc334bb0e80f6aae07ad9c0cd41d0b"} err="failed to get container status \"638d392f0090b7eda6efe0cd05dccb1448cc334bb0e80f6aae07ad9c0cd41d0b\": rpc error: code = NotFound desc = could not find container \"638d392f0090b7eda6efe0cd05dccb1448cc334bb0e80f6aae07ad9c0cd41d0b\": container with ID starting with 638d392f0090b7eda6efe0cd05dccb1448cc334bb0e80f6aae07ad9c0cd41d0b not found: ID does not exist"
Jul 10 19:21:18.132764 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.132719    2565 scope.go:117] "RemoveContainer" containerID="8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8"
Jul 10 19:21:18.136798 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.136775    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"]
Jul 10 19:21:18.141787 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.141766    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6bj4h4w"]
Jul 10 19:21:18.141787 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.141778    2565 scope.go:117] "RemoveContainer" containerID="8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45"
Jul 10 19:21:18.155322 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.155294    2565 scope.go:117] "RemoveContainer" containerID="96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4"
Jul 10 19:21:18.156063 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.156043    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"]
Jul 10 19:21:18.159344 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.159322    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6dcf5c9d69c2sbm"]
Jul 10 19:21:18.165599 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.165579    2565 scope.go:117] "RemoveContainer" containerID="8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8"
Jul 10 19:21:18.165933 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:21:18.165907    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8\": container with ID starting with 8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8 not found: ID does not exist" containerID="8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8"
Jul 10 19:21:18.166004 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.165947    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8"} err="failed to get container status \"8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8\": rpc error: code = NotFound desc = could not find container \"8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8\": container with ID starting with 8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8 not found: ID does not exist"
Jul 10 19:21:18.166004 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.165974    2565 scope.go:117] "RemoveContainer" containerID="8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45"
Jul 10 19:21:18.166296 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:21:18.166272    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45\": container with ID starting with 8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45 not found: ID does not exist" containerID="8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45"
Jul 10 19:21:18.166409 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.166299    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45"} err="failed to get container status \"8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45\": rpc error: code = NotFound desc = could not find container \"8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45\": container with ID starting with 8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45 not found: ID does not exist"
Jul 10 19:21:18.166409 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.166315    2565 scope.go:117] "RemoveContainer" containerID="96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4"
Jul 10 19:21:18.166551 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:21:18.166533    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4\": container with ID starting with 96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4 not found: ID does not exist" containerID="96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4"
Jul 10 19:21:18.166608 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.166554    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4"} err="failed to get container status \"96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4\": rpc error: code = NotFound desc = could not find container \"96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4\": container with ID starting with 96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4 not found: ID does not exist"
Jul 10 19:21:18.166608 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.166569    2565 scope.go:117] "RemoveContainer" containerID="8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8"
Jul 10 19:21:18.166839 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.166813    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8"} err="failed to get container status \"8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8\": rpc error: code = NotFound desc = could not find container \"8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8\": container with ID starting with 8863000204e8aa1716af9634c4ec67e645a4fe40794cef5062ea0cf0f73827f8 not found: ID does not exist"
Jul 10 19:21:18.166912 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.166839    2565 scope.go:117] "RemoveContainer" containerID="8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45"
Jul 10 19:21:18.167068 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.167050    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45"} err="failed to get container status \"8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45\": rpc error: code = NotFound desc = could not find container \"8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45\": container with ID starting with 8d8736da4004d7ae3a9433e1947aa56819343d0b2a88d985f7916260086c5d45 not found: ID does not exist"
Jul 10 19:21:18.167137 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.167068    2565 scope.go:117] "RemoveContainer" containerID="96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4"
Jul 10 19:21:18.167305 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:18.167280    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4"} err="failed to get container status \"96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4\": rpc error: code = NotFound desc = could not find container \"96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4\": container with ID starting with 96ba7c3f9f43eac2ba75bdc030f935faf9995f596cfbe96a42d83db53af112b4 not found: ID does not exist"
Jul 10 19:21:19.098719 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:19.098668    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" path="/var/lib/kubelet/pods/0aea6ed6-19f0-4664-a0ca-bf457a5275d9/volumes"
Jul 10 19:21:19.099492 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:19.099461    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" path="/var/lib/kubelet/pods/34f9ebfe-9ece-463f-a37e-5091e0954c56/volumes"
Jul 10 19:21:22.093560 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.093530    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 10 19:21:22.093997 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.093731    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main" containerID="cri-o://227a15a8d829a2d0f1665d5c4025f612ef701d1fc6e08b58b36dd31b6ac2cbb3" gracePeriod=30
Jul 10 19:21:22.836898 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.836870    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:21:22.869264 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.869231    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-home\") pod \"70c35f1a-1af5-4d58-897c-f74d09514f26\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") "
Jul 10 19:21:22.869264 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.869277    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhbq5\" (UniqueName: \"kubernetes.io/projected/70c35f1a-1af5-4d58-897c-f74d09514f26-kube-api-access-bhbq5\") pod \"70c35f1a-1af5-4d58-897c-f74d09514f26\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") "
Jul 10 19:21:22.869531 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.869305    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-dshm\") pod \"70c35f1a-1af5-4d58-897c-f74d09514f26\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") "
Jul 10 19:21:22.869531 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.869339    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-kserve-provision-location\") pod \"70c35f1a-1af5-4d58-897c-f74d09514f26\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") "
Jul 10 19:21:22.869531 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.869393    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c35f1a-1af5-4d58-897c-f74d09514f26-tls-certs\") pod \"70c35f1a-1af5-4d58-897c-f74d09514f26\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") "
Jul 10 19:21:22.869531 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.869467    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-tmp-dir\") pod \"70c35f1a-1af5-4d58-897c-f74d09514f26\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") "
Jul 10 19:21:22.869531 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.869491    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-model-cache\") pod \"70c35f1a-1af5-4d58-897c-f74d09514f26\" (UID: \"70c35f1a-1af5-4d58-897c-f74d09514f26\") "
Jul 10 19:21:22.869925 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.869883    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-model-cache" (OuterVolumeSpecName: "model-cache") pod "70c35f1a-1af5-4d58-897c-f74d09514f26" (UID: "70c35f1a-1af5-4d58-897c-f74d09514f26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:22.870055 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.870007    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-home" (OuterVolumeSpecName: "home") pod "70c35f1a-1af5-4d58-897c-f74d09514f26" (UID: "70c35f1a-1af5-4d58-897c-f74d09514f26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:22.871627 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.871583    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-dshm" (OuterVolumeSpecName: "dshm") pod "70c35f1a-1af5-4d58-897c-f74d09514f26" (UID: "70c35f1a-1af5-4d58-897c-f74d09514f26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:22.871733 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.871680    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c35f1a-1af5-4d58-897c-f74d09514f26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70c35f1a-1af5-4d58-897c-f74d09514f26" (UID: "70c35f1a-1af5-4d58-897c-f74d09514f26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:21:22.871940 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.871918    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c35f1a-1af5-4d58-897c-f74d09514f26-kube-api-access-bhbq5" (OuterVolumeSpecName: "kube-api-access-bhbq5") pod "70c35f1a-1af5-4d58-897c-f74d09514f26" (UID: "70c35f1a-1af5-4d58-897c-f74d09514f26"). InnerVolumeSpecName "kube-api-access-bhbq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:21:22.881835 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.881805    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70c35f1a-1af5-4d58-897c-f74d09514f26" (UID: "70c35f1a-1af5-4d58-897c-f74d09514f26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:22.931337 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.931233    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70c35f1a-1af5-4d58-897c-f74d09514f26" (UID: "70c35f1a-1af5-4d58-897c-f74d09514f26"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:22.970955 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.970924    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-dshm\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:22.970955 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.970953    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-kserve-provision-location\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:22.970955 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.970962    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c35f1a-1af5-4d58-897c-f74d09514f26-tls-certs\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:22.971231 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.970973    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-tmp-dir\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:22.971231 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.970981    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-model-cache\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:22.971231 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.970989    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c35f1a-1af5-4d58-897c-f74d09514f26-home\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:22.971231 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:22.970997    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhbq5\" (UniqueName: \"kubernetes.io/projected/70c35f1a-1af5-4d58-897c-f74d09514f26-kube-api-access-bhbq5\") on node \"ip-10-0-132-91.ec2.internal\" DevicePath \"\""
Jul 10 19:21:23.144363 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:23.144312    2565 generic.go:356] "Generic (PLEG): container finished" podID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerID="227a15a8d829a2d0f1665d5c4025f612ef701d1fc6e08b58b36dd31b6ac2cbb3" exitCode=0
Jul 10 19:21:23.144802 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:23.144425    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jul 10 19:21:23.144802 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:23.144411    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"70c35f1a-1af5-4d58-897c-f74d09514f26","Type":"ContainerDied","Data":"227a15a8d829a2d0f1665d5c4025f612ef701d1fc6e08b58b36dd31b6ac2cbb3"}
Jul 10 19:21:23.144802 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:23.144614    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"70c35f1a-1af5-4d58-897c-f74d09514f26","Type":"ContainerDied","Data":"540faafb82021419e4b8d126accf7aa4800953284241d1c5d90849398b2f0cbf"}
Jul 10 19:21:23.144802 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:23.144669    2565 scope.go:117] "RemoveContainer" containerID="227a15a8d829a2d0f1665d5c4025f612ef701d1fc6e08b58b36dd31b6ac2cbb3"
Jul 10 19:21:23.153999 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:23.153977    2565 scope.go:117] "RemoveContainer" containerID="fca178949bb037af18b8517f52b03af5ed92b41e2db8869a7957a4f971f1e915"
Jul 10 19:21:23.163989 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:23.163963    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 10 19:21:23.168641 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:23.168616    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jul 10 19:21:23.218818 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:23.218794    2565 scope.go:117] "RemoveContainer" containerID="227a15a8d829a2d0f1665d5c4025f612ef701d1fc6e08b58b36dd31b6ac2cbb3"
Jul 10 19:21:23.219190 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:21:23.219166    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"227a15a8d829a2d0f1665d5c4025f612ef701d1fc6e08b58b36dd31b6ac2cbb3\": container with ID starting with 227a15a8d829a2d0f1665d5c4025f612ef701d1fc6e08b58b36dd31b6ac2cbb3 not found: ID does not exist" containerID="227a15a8d829a2d0f1665d5c4025f612ef701d1fc6e08b58b36dd31b6ac2cbb3"
Jul 10 19:21:23.219237 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:23.219204    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"227a15a8d829a2d0f1665d5c4025f612ef701d1fc6e08b58b36dd31b6ac2cbb3"} err="failed to get container status \"227a15a8d829a2d0f1665d5c4025f612ef701d1fc6e08b58b36dd31b6ac2cbb3\": rpc error: code = NotFound desc = could not find container \"227a15a8d829a2d0f1665d5c4025f612ef701d1fc6e08b58b36dd31b6ac2cbb3\": container with ID starting with 227a15a8d829a2d0f1665d5c4025f612ef701d1fc6e08b58b36dd31b6ac2cbb3 not found: ID does not exist"
Jul 10 19:21:23.219237 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:23.219224    2565 scope.go:117] "RemoveContainer" containerID="fca178949bb037af18b8517f52b03af5ed92b41e2db8869a7957a4f971f1e915"
Jul 10 19:21:23.219483 ip-10-0-132-91 kubenswrapper[2565]: E0710 19:21:23.219464    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca178949bb037af18b8517f52b03af5ed92b41e2db8869a7957a4f971f1e915\": container with ID starting with fca178949bb037af18b8517f52b03af5ed92b41e2db8869a7957a4f971f1e915 not found: ID does not exist" containerID="fca178949bb037af18b8517f52b03af5ed92b41e2db8869a7957a4f971f1e915"
Jul 10 19:21:23.219522 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:23.219490    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca178949bb037af18b8517f52b03af5ed92b41e2db8869a7957a4f971f1e915"} err="failed to get container status \"fca178949bb037af18b8517f52b03af5ed92b41e2db8869a7957a4f971f1e915\": rpc error: code = NotFound desc = could not find container \"fca178949bb037af18b8517f52b03af5ed92b41e2db8869a7957a4f971f1e915\": container with ID starting with fca178949bb037af18b8517f52b03af5ed92b41e2db8869a7957a4f971f1e915 not found: ID does not exist"
Jul 10 19:21:25.095610 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:21:25.095571    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" path="/var/lib/kubelet/pods/70c35f1a-1af5-4d58-897c-f74d09514f26/volumes"
Jul 10 19:24:02.528811 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:02.528782    2565 ???:1] "http2: server: error reading preface from client 10.0.132.91:44260: read tcp 10.0.132.91:10250->10.0.132.91:44260: read: connection reset by peer"
Jul 10 19:24:02.554077 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:02.554050    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-pqfjx_84b2d65b-d93f-4385-82d5-3e1df7ab3e97/authorino/0.log"
Jul 10 19:24:02.583270 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:02.583240    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-zcjhd_71ee2e20-47f1-4ef6-aafd-ee4d850840d4/manager/0.log"
Jul 10 19:24:02.757768 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:02.757743    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj_b421cb67-e157-41d4-b071-5247ed017ff6/manager/0.log"
Jul 10 19:24:05.134818 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.134787    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l54pc/must-gather-ztxhf"]
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135036    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="storage-initializer"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135046    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="storage-initializer"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135058    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="llm-d-routing-sidecar"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135064    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="llm-d-routing-sidecar"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135071    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135076    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135082    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135088    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135095    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="storage-initializer"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135101    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="storage-initializer"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135112    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="main"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135118    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="main"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135125    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135130    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135139    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="storage-initializer"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135157    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="storage-initializer"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135165    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="storage-initializer"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135170    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="storage-initializer"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135212    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="70c35f1a-1af5-4d58-897c-f74d09514f26" containerName="main"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135220    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="llm-d-routing-sidecar"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135225    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1571e38c-c6f2-40a4-a5e8-a29d9ab9c919" containerName="main"
Jul 10 19:24:05.135220 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135230    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aea6ed6-19f0-4664-a0ca-bf457a5275d9" containerName="main"
Jul 10 19:24:05.135913 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.135237    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="34f9ebfe-9ece-463f-a37e-5091e0954c56" containerName="main"
Jul 10 19:24:05.138115 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.138095    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l54pc/must-gather-ztxhf"
Jul 10 19:24:05.140732 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.140710    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l54pc\"/\"openshift-service-ca.crt\""
Jul 10 19:24:05.141984 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.141967    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l54pc\"/\"kube-root-ca.crt\""
Jul 10 19:24:05.142073 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.141967    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l54pc\"/\"default-dockercfg-zmwvt\""
Jul 10 19:24:05.146853 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.146826    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l54pc/must-gather-ztxhf"]
Jul 10 19:24:05.210885 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.210854    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d197b21-e75e-4af4-8416-335893cb713a-must-gather-output\") pod \"must-gather-ztxhf\" (UID: \"3d197b21-e75e-4af4-8416-335893cb713a\") " pod="openshift-must-gather-l54pc/must-gather-ztxhf"
Jul 10 19:24:05.211056 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.210907    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wnk6\" (UniqueName: \"kubernetes.io/projected/3d197b21-e75e-4af4-8416-335893cb713a-kube-api-access-9wnk6\") pod \"must-gather-ztxhf\" (UID: \"3d197b21-e75e-4af4-8416-335893cb713a\") " pod="openshift-must-gather-l54pc/must-gather-ztxhf"
Jul 10 19:24:05.311789 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.311752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d197b21-e75e-4af4-8416-335893cb713a-must-gather-output\") pod \"must-gather-ztxhf\" (UID: \"3d197b21-e75e-4af4-8416-335893cb713a\") " pod="openshift-must-gather-l54pc/must-gather-ztxhf"
Jul 10 19:24:05.311789 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.311792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wnk6\" (UniqueName: \"kubernetes.io/projected/3d197b21-e75e-4af4-8416-335893cb713a-kube-api-access-9wnk6\") pod \"must-gather-ztxhf\" (UID: \"3d197b21-e75e-4af4-8416-335893cb713a\") " pod="openshift-must-gather-l54pc/must-gather-ztxhf"
Jul 10 19:24:05.312179 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.312121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d197b21-e75e-4af4-8416-335893cb713a-must-gather-output\") pod \"must-gather-ztxhf\" (UID: \"3d197b21-e75e-4af4-8416-335893cb713a\") " pod="openshift-must-gather-l54pc/must-gather-ztxhf"
Jul 10 19:24:05.324416 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.324385    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wnk6\" (UniqueName: \"kubernetes.io/projected/3d197b21-e75e-4af4-8416-335893cb713a-kube-api-access-9wnk6\") pod \"must-gather-ztxhf\" (UID: \"3d197b21-e75e-4af4-8416-335893cb713a\") " pod="openshift-must-gather-l54pc/must-gather-ztxhf"
Jul 10 19:24:05.447719 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.447646    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l54pc/must-gather-ztxhf"
Jul 10 19:24:05.578229 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.578196    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l54pc/must-gather-ztxhf"]
Jul 10 19:24:05.580331 ip-10-0-132-91 kubenswrapper[2565]: W0710 19:24:05.580297    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d197b21_e75e_4af4_8416_335893cb713a.slice/crio-08647df2f55563624b04073f80b627a685307b45e1634e9d8b3296a94122dae7 WatchSource:0}: Error finding container 08647df2f55563624b04073f80b627a685307b45e1634e9d8b3296a94122dae7: Status 404 returned error can't find the container with id 08647df2f55563624b04073f80b627a685307b45e1634e9d8b3296a94122dae7
Jul 10 19:24:05.582124 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.582104    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:24:05.798838 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:05.798798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l54pc/must-gather-ztxhf" event={"ID":"3d197b21-e75e-4af4-8416-335893cb713a","Type":"ContainerStarted","Data":"08647df2f55563624b04073f80b627a685307b45e1634e9d8b3296a94122dae7"}
Jul 10 19:24:06.806265 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:06.806034    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l54pc/must-gather-ztxhf" event={"ID":"3d197b21-e75e-4af4-8416-335893cb713a","Type":"ContainerStarted","Data":"d160920d85b9740a49b0afb8ea72d60c04921bff484da16e32013ff87cf44108"}
Jul 10 19:24:07.812976 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:07.812943    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l54pc/must-gather-ztxhf" event={"ID":"3d197b21-e75e-4af4-8416-335893cb713a","Type":"ContainerStarted","Data":"68b5f8629610d646a9652ffecae8f00e42c28dd212a4929f51fed74dca1d041e"}
Jul 10 19:24:07.832327 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:07.832269    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l54pc/must-gather-ztxhf" podStartSLOduration=1.790447283 podStartE2EDuration="2.832248742s" totalImagesPullingTime="1.041801459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:24:07.830662093 +0000 UTC m=+2625.298580926" watchObservedRunningTime="2026-07-10 19:24:07.832248742 +0000 UTC m=+2625.300167575"
Jul 10 19:24:08.268267 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:08.268236    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sljz9_d6730ada-8f92-421c-8768-422891b4c4f9/global-pull-secret-syncer/0.log"
Jul 10 19:24:08.346970 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:08.346936    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nkk6q_344d8617-d6ba-4abe-a743-a638e7f0bfad/konnectivity-agent/0.log"
Jul 10 19:24:08.405674 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:08.405642    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-91.ec2.internal_b3d2324b5b6efc54e3cd149d6b3cc67c/haproxy/0.log"
Jul 10 19:24:13.468691 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:13.468654    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-pqfjx_84b2d65b-d93f-4385-82d5-3e1df7ab3e97/authorino/0.log"
Jul 10 19:24:13.553022 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:13.552991    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-zcjhd_71ee2e20-47f1-4ef6-aafd-ee4d850840d4/manager/0.log"
Jul 10 19:24:13.671870 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:13.671833    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-tp6kj_b421cb67-e157-41d4-b071-5247ed017ff6/manager/0.log"
Jul 10 19:24:15.185602 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:15.185567    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-vfrhr_8d0f4fd3-5c15-4f46-b466-5bed333a7dc7/kube-state-metrics/0.log"
Jul 10 19:24:15.208614 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:15.208583    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-vfrhr_8d0f4fd3-5c15-4f46-b466-5bed333a7dc7/kube-rbac-proxy-main/0.log"
Jul 10 19:24:15.239432 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:15.239400    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-vfrhr_8d0f4fd3-5c15-4f46-b466-5bed333a7dc7/kube-rbac-proxy-self/0.log"
Jul 10 19:24:15.320929 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:15.320860    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-6vpv7_33fdee38-87f5-4255-8f13-5ba6bae49a83/monitoring-plugin/0.log"
Jul 10 19:24:15.361909 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:15.361880    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29gsc_914043ab-d211-44c3-9376-f5d6f3d40374/node-exporter/0.log"
Jul 10 19:24:15.389054 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:15.389030    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29gsc_914043ab-d211-44c3-9376-f5d6f3d40374/kube-rbac-proxy/0.log"
Jul 10 19:24:15.413340 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:15.413308    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29gsc_914043ab-d211-44c3-9376-f5d6f3d40374/init-textfile/0.log"
Jul 10 19:24:17.413285 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:17.413253    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-fm74x_2e81b456-090c-4ce7-832a-1d57a18344cb/networking-console-plugin/0.log"
Jul 10 19:24:17.791509 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:17.791470    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"]
Jul 10 19:24:17.796707 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:17.796673    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:17.812505 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:17.812463    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"]
Jul 10 19:24:17.931796 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:17.931750    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-podres\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:17.931990 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:17.931831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-proc\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:17.931990 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:17.931863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-sys\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:17.931990 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:17.931888    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddb5z\" (UniqueName: \"kubernetes.io/projected/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-kube-api-access-ddb5z\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:17.931990 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:17.931936    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-lib-modules\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:18.033334 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.033279    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-proc\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:18.033737 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.033348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-sys\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:18.033737 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.033379    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddb5z\" (UniqueName: \"kubernetes.io/projected/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-kube-api-access-ddb5z\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:18.033737 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.033422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-lib-modules\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:18.033737 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.033493    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-podres\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:18.033737 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.033667    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-podres\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:18.033737 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.033728    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-proc\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:18.034088 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.033773    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-sys\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:18.034228 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.034206    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-lib-modules\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:18.044584 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.044478    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddb5z\" (UniqueName: \"kubernetes.io/projected/ffc7c328-ce56-4d5d-b25d-47dc1de8aab4-kube-api-access-ddb5z\") pod \"perf-node-gather-daemonset-r7f7k\" (UID: \"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4\") " pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:18.108904 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.108872    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:18.265247 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.265137    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"]
Jul 10 19:24:18.268478 ip-10-0-132-91 kubenswrapper[2565]: W0710 19:24:18.268453    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podffc7c328_ce56_4d5d_b25d_47dc1de8aab4.slice/crio-1be8bc51c8960beb28c27c7d9c20866052792c0d503102e11a03e7eeec92d22c WatchSource:0}: Error finding container 1be8bc51c8960beb28c27c7d9c20866052792c0d503102e11a03e7eeec92d22c: Status 404 returned error can't find the container with id 1be8bc51c8960beb28c27c7d9c20866052792c0d503102e11a03e7eeec92d22c
Jul 10 19:24:18.877693 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.877662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k" event={"ID":"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4","Type":"ContainerStarted","Data":"b036a756bcbde2ef0b8a2b23a9b1c18b4cc66fea4ae67cea92d091ce24e7cc4e"}
Jul 10 19:24:18.878072 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.877716    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k" event={"ID":"ffc7c328-ce56-4d5d-b25d-47dc1de8aab4","Type":"ContainerStarted","Data":"1be8bc51c8960beb28c27c7d9c20866052792c0d503102e11a03e7eeec92d22c"}
Jul 10 19:24:18.878072 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.877750    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:18.901815 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:18.901747    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k" podStartSLOduration=1.9017241409999999 podStartE2EDuration="1.901724141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:24:18.90164761 +0000 UTC m=+2636.369566442" watchObservedRunningTime="2026-07-10 19:24:18.901724141 +0000 UTC m=+2636.369642975"
Jul 10 19:24:20.232280 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:20.232250    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gp9v9_52c473ba-b012-460e-89a1-e32d7b7dc013/dns/0.log"
Jul 10 19:24:20.254662 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:20.254631    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gp9v9_52c473ba-b012-460e-89a1-e32d7b7dc013/kube-rbac-proxy/0.log"
Jul 10 19:24:20.381005 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:20.380976    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vb9ww_48f95840-5a3c-4014-b1c5-f489153c94ce/dns-node-resolver/0.log"
Jul 10 19:24:21.024197 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:21.024165    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bzjdq_975a4c98-91a8-4277-b3ff-a5cd2da229cb/node-ca/0.log"
Jul 10 19:24:22.667771 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:22.667738    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8zfrf_6d6e03d6-117c-4854-b76c-dba72a1a2f27/serve-healthcheck-canary/0.log"
Jul 10 19:24:23.265886 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:23.265850    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f4w65_2f43cba3-15f4-4170-8b22-36f666bffe9d/kube-rbac-proxy/0.log"
Jul 10 19:24:23.285933 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:23.285904    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f4w65_2f43cba3-15f4-4170-8b22-36f666bffe9d/exporter/0.log"
Jul 10 19:24:23.308239 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:23.308206    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f4w65_2f43cba3-15f4-4170-8b22-36f666bffe9d/extractor/0.log"
Jul 10 19:24:24.893770 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:24.893741    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l54pc/perf-node-gather-daemonset-r7f7k"
Jul 10 19:24:26.257183 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:26.257135    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-849cc6f9d7-gwqhq_416e9ca7-99be-40d1-aba0-dd3ff1ecee2a/manager/0.log"
Jul 10 19:24:27.455796 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:27.455704    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-89f8q_c6097c37-d993-4954-a077-b8b6529a3c1b/s3-init/0.log"
Jul 10 19:24:34.647628 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:34.647601    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvx67_be7bd63d-2e6e-4570-abf3-272865bfef5e/kube-multus-additional-cni-plugins/0.log"
Jul 10 19:24:34.669256 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:34.669223    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvx67_be7bd63d-2e6e-4570-abf3-272865bfef5e/egress-router-binary-copy/0.log"
Jul 10 19:24:34.689271 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:34.689244    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvx67_be7bd63d-2e6e-4570-abf3-272865bfef5e/cni-plugins/0.log"
Jul 10 19:24:34.710049 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:34.710024    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvx67_be7bd63d-2e6e-4570-abf3-272865bfef5e/bond-cni-plugin/0.log"
Jul 10 19:24:34.731583 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:34.731557    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvx67_be7bd63d-2e6e-4570-abf3-272865bfef5e/routeoverride-cni/0.log"
Jul 10 19:24:34.751340 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:34.751295    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvx67_be7bd63d-2e6e-4570-abf3-272865bfef5e/whereabouts-cni-bincopy/0.log"
Jul 10 19:24:34.771872 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:34.771834    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvx67_be7bd63d-2e6e-4570-abf3-272865bfef5e/whereabouts-cni/0.log"
Jul 10 19:24:34.848633 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:34.848596    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jfz8j_8f165c26-834e-4a90-b142-8293f6860d4f/kube-multus/0.log"
Jul 10 19:24:35.011508 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:35.011475    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dj7xt_b90039f7-1ab3-4de0-81fe-4829c384ed89/network-metrics-daemon/0.log"
Jul 10 19:24:35.035895 ip-10-0-132-91 kubenswrapper[2565]: I0710 19:24:35.035867    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dj7xt_b90039f7-1ab3-4de0-81fe-4829c384ed89/kube-rbac-proxy/0.log"